diff --git a/docs/MODEL_FRAMEWORK.md b/docs/MODEL_FRAMEWORK.md new file mode 100644 index 000000000..d628c1a1b --- /dev/null +++ b/docs/MODEL_FRAMEWORK.md @@ -0,0 +1,410 @@ +# Model framework (r3) + +A plan for moving Dark Cloud's models, characters and animations to glTF 2.0, converted at +extraction and loaded natively, as ChronicleTwo does for Dark Cloud 2: models become ordinary glTF +files that tools such as Blender open, animations become named actions, and a datapack can replace +a model with its own glTF. Cloth keeps its own section (section 8), with an optional Enhanced mode +that keeps it out of the body. + +What isn't particular to Dark Cloud goes into Kazarov, as for audio (docs/AUDIO_FRAMEWORK.md): the +converter and the motion name table. Chronicle keeps what only it has: its character configuration +commands, its glTF loader, its cloth and the hooks from its retail loaders. The glTF +loader is Chronicle's own for now (section 5); it can be unified with ChronicleTwo's once both games +are ready. + +The four phases are built; section 12 says where each stands and what is left for later. +Sections 2 and 3 describe what existed and what was measured before. + +--- + +## 1. Goals + +- **Open models.** Every model on the disc, characters, weapons, items, scenes and georama parts + alike, published as glTF 2.0, with its meshes, materials, textures, skeleton, skin weights and morphs, and its motions as + named actions in English. +- **The game unchanged.** The converted data plays exactly as the retail files do: the same poses + at the same frame numbers, the same attachment points, hit windows and events. Retail code keeps + addressing motions by frame number, as it does today. +- **Moddable.** A datapack can replace any model with its own glTF, made in a standard tool. +- **Shared.** The converter and the general glTF loading live in Kazarov and serve both games. +- **Cloth that can be better.** Dark Cloud's cloth data converts as it is, and an optional mode keeps + the cloth out of the arms and legs (section 8). + +--- + +## 2. Today + +**Formats.** Dark Cloud and Dark Cloud 2 share Level-5's engine formats: + +| Format | Holds | Dark Cloud loader | +|---|---|---| +| MDS | a scene: one record per object (name, parent, local matrix, offset of its MDT) | `LoadMDSFile` (`dataset.cpp`) | +| MDT | a mesh inside an MDS: positions, normals, colours, UVs, strips with per-corner indices, materials | `CVisualVu1::CreateVUdataFromMDT` (`port/src/game/visualvu1.cpp`) | +| MOT | keyframe channels: rotation, scale, translation, vertex morphs, camera, material, visibility | `CreateAnimeDataEX` (`gameutil.cpp`) | +| WGT | skin weights in the same channel format: per mesh frame, bone and vertex, a percentage | `CreateAnimeDataEX` | +| BBP | bind-pose matrices, one per frame | `CreateAnimeDataEX` | +| CHR | a pack of the above with a character `.cfg` and textures | `CCharacter::LoadPackData*`, `ReadInfo` (`chararead.cpp`) | +| CLO | cloth parameters (section 8) | `InitCloth` (`clothread.cpp`) | + +The record layouts match what Kazarov's converter parses field for field: MDS objects (name at 8, +MDT at 40, parent at 44, matrix at 48), the MDT header's count and offset pairs, 96-byte materials, +and the 32-byte MOT and WGT channel and key records with the same type numbers. One difference is +known: Dark Cloud's material texture name runs 44 bytes where Dark Cloud 2 reads 32 and then flags. + +**The port today** uses the retail files in place. `LoadMDSFile` is not replaced; meshes are built +for Kazarov's renderer from the MDT (`visualvu1.cpp`), skinning runs on the CPU (`MotionProc2`, +`port/src/game/gameutil.cpp`), and the display list interpolates frame matrices and rewritten vertices +between ticks. Kazarov's GPU skinning path (prog0 palettes) is Dark Cloud 2's and unused here. + +**What reads models.** About 92 calls of `LoadMDSFile` (half in the title and opening), about 113 +of `LoadPackData*`, 14 of `LoadCollisionFile`, and 4 of `CMotionModel::LoadPack`. Game logic finds +frames by name (`SearchFrame`, about 150 calls: `weapon`, `r_foot`, `dcol0`...), attaches weapons +(`SetReference`), and compares a character's motion time with absolute frame numbers: a hit window +is `time >= 674 && time <= 677` (`charakey.cpp`). So frame numbers are part of the game's logic. + +**Kazarov's converter** (`tools/kazarov/src/rust/models.rs`, 8,475 lines; `actions.rs`, 503) +publishes Dark Cloud 2's models as glTF: nodes from MDS objects, meshes with skin joints and weights, +inverse binds from BBP, morph targets from vertex channels, PBR materials with the converted +textures, animations sampled at 30 frames per second, and a `.game.json` sidecar holding only the +game semantics glTF can't express. Actions are named through `assets/motion_names.json` (2,870 +entries). Its parsers carry no version checks; the Dark Cloud 2 specific parts are the player-unit +tables, external motion placement from DC2's scripts, DC2's disc layout and the motion table. The C++ +bridge exposes no model function yet. + +**ChronicleTwo's runtime** (`port/src/platform/model_assets.cpp`, about 3,700 lines) reads the glTF +with fastgltf and fills the retail structures directly (frames, mesh visuals, motion lists), through +about a dozen replaced loaders. Animations are evaluated natively from the glTF samplers, each action +given a range of game frames. Its general parts (document and datapack reading, accessors, skins, +sampling) are a few hundred lines written against Dark Cloud 2's structures. + +--- + +## 3. Measured + +On the PAL disc's extracted data: + +| | Count | Size | +|---|---|---| +| Character packs (`.chr`) | 1,116 | 495 MB | +| Models (`.mds`), loose and in packs | 412 and 5,113 | 6 and 148 MB | +| Motions (`.mot`), in packs | 2,455 | 307 MB | +| Skin weights (`.wgt`) and bind poses (`.bbp`) | 1,360 and 1,424 | 36 and 4 MB | +| Cloth (`.clo`) | 104 in packs, 7 loose | small | +| Character `.cfg` files | 1,265, with 9,573 `KEY` lines | | + +**Kazarov's converter, unchanged, on Dark Cloud's packs.** Its own `retail_pack_smoke` test converts +Toan's pack (`chara/c01d.chr`) and a monster's (`chara/f01a.chr`) and passes its validation: Toan's +462 nodes, 26 meshes and 210 primitives, the monster's skeleton and skin. Three things don't come +across yet, all from Dark Cloud's character `.cfg`: + +- **Actions.** A Dark Cloud motion file is one timeline cut into actions by `KEY start, end, speed` + lines, named only in a trailing comment: `KEY 10, 20, 0.05 //立ち 0` (standing). The converter + doesn't read that syntax, so each file becomes one animation. 9,419 of the 9,573 keys carry a + name, 2,391 distinct; Dark Cloud 2's table already translates 237 of them, covering 36% of keys. +- **Toan's skin.** `MOTION 0, "c01dbas.mot", "c01dbas.bbp", "c01dbas.wgt"` names the three files + together; Toan converts without a skin, likely because this form isn't read. +- **Dark Cloud's own commands:** `TEX_ANIME` (Toan's blinking eyes), `MATERIAL_ANIME`, + `VERTEX_ANIME`, `BODY_SIZE`, `ALLOC_MDT`, `ALLOC_DBUFF`, `FOOT`, `EVENT`, `CLOTH`. + +--- + +## 4. Extraction + +`dcdata` converts the models through Kazarov's converter, the one Dark Cloud 2 uses, with a **game +profile** saying how Dark Cloud's data differs: + +- **Kazarov** (`models_convert_dc1`) converts the textures and then every model, motion, weight + and bind pose into one tree, so each glTF points at the PNGs beside it. The profile carries: + - **the game's configuration reader.** Chronicle hands Kazarov a reader (`ProfileReader`) that + turns a `.cfg` or `.clo` into typed commands; Kazarov doesn't know Dark Cloud's syntax. + - **absolute frames.** Actions and `KEY` commands keep their source frames. + - **fixed strip streams.** Every Dark Cloud strip corner indexes a position, normal and texture + coordinate, plus a colour when the mesh has colours, as `CVisualVu1::CreateVUdataFromMDT` + reads them; Dark Cloud 2's prim-word flags don't apply. + - **source records.** Sidecars keep what rebuilding each source file byte for byte needs (its + header and reserved words, local matrices, whole material records, the words no reader looks + at), and `data/models.json` indexes every source by its retail name. +- **Chronicle** (`tools/dcdata/character_profile.hpp`) reads `.cfg` and `.clo` word for word as + `ReadInfo`, `CTextureAnime::LoadCFGFile` and `InitCloth` do. A `KEY`'s name is the comment on its + line, without the index after it (`KEY 10, 20, 0.05 //立ち 0`), and becomes Dark Cloud 2's + form, `KEY "立ち", 10, 20, 0.05`. A comment that is only the index leaves the key unnamed; it + takes its motion's name. + +Dark Cloud's data needed a few things the converter now handles for any game: + +- **Collision meshes.** An MDT counting no texture coordinates holds collision triangles + (`MDT_COLLISION`: vertex indices, an attribute index and a reserved word), as every such MDT on + the disc does. It becomes a triangle mesh, its attribute records kept in the sidecar. +- **Indices past a stream's count.** Some strips and skins index a vertex past its stream's count + that the mesh still holds; the converter reads it from the mesh's own bytes, as the game does. +- **Unused key lanes** whose bits aren't numbers keep their words. +- **Animated memory card icons**: the other shapes become morph targets. + +| Disc | Published | +|---|---| +| a character pack, `chara/c01d.chr` | `chara/c01d.gltf` (+ `.gltf.bin`) and `c01d.game.json`, the sidecar, which also holds its configuration as typed commands | +| a loose model, `dun/item/main_data/.mds` | `.gltf` and a minimal `.game.json` | +| a cloth file | its typed commands in the character's sidecar (section 8) | +| textures a model uses | the PNGs the texture bake already writes, referenced from the glTF | + +**Actions keep their frames.** Each `KEY` becomes one named glTF action, and the sidecar records its +original start and end frame. The runtime places each action back at those frames, so retail code +that asks for frame 674 still gets what the motion file holds at 674. Unlike ChronicleTwo, which +gives actions fresh frame ranges, Dark Cloud's absolute hit windows and event frames depend on this. + +**Names.** An action's English name comes from Kazarov's shared table by its Japanese comment name +(`立ち` → `idle`). Dark Cloud adds 2,293 names to that table (5,163 now), translated for what the +motion is for in the game rather than word for word (`釣り上げられ中` is `being_reeled_in`). A +motion-slot number before a name becomes a suffix (`0立ち` → `idle_0`). An unnamed key takes its +motion's name (`c01dbas`, `c01dbas_v2`...). + +**Checks** run on the real disc (section 10): every model, weight and bind pose rebuilt from its +bundle equals the disc's bytes, and every character motion keeps its poses in every action. + +**Measured** on the PAL disc: all 5,166 files holding models convert, in about four and a half +minutes; 11,782 sources are indexed (6,737 models, 1,953 character motions, 1,409 bind poses, 1,342 +weights). Kazarov places a motion pack's motions into every model they fit, as for Dark Cloud 2; +322 fit none and keep their retail keys. + +--- + +## 5. Loading + +The game's own loaders (`LoadMDSFile`, `LoadCollisionFile`, `CreateAnimeDataEX`, cloth, vertex +animation) stay as they are, and so do their 200-odd callers. What changes is where their records +come from: `port/src/data/model_gltf.cpp` rebuilds them from the published bundles. + +- **When the game reads a file**, every published model, motion, weight and bind pose inside it, + in nested packs and in scenes at offsets alike, is rebuilt from its bundle and written into the + game's buffer where the disc has it. The game then edits it in place as it does on the PS2 (shadow + edge flags, for one), and the bytes live as long as the buffer does. +- **Models, weights and bind poses** come back byte for byte: the glTF holds the geometry and the + sidecar the rest. +- **Motions** come back from their glTF actions: each action's clip gives the transform keys inside + it at their source frames. The sidecar keeps what a clip can't give back: the keys outside every + action and on each action's first and last frame, the channels glTF has no place for (morphs, + camera, material, visibility), and the rotation and scale keys of frames the converter rebaked + from the pose. Every motion keeps the disc's key frames, so none grows past the disc's, whose + sizes the game's arenas are made for. +- Frames keep their names, attributes and order, so `SearchFrame`, attachments and hit windows work + as before. + +Georama scenes (`.scn`), dungeon map parts (`.mpd`), the opening's scenes (`.sne`) and the title load +the same way: the converter finds their scenes, and the rebuild puts each back at its offset. + +**Chronicle's own loader.** The glTF is read with fastgltf, which Kazarov already builds, by a loader +in Chronicle's `port/src`. ChronicleTwo's loader does the same +for Dark Cloud 2's; the general parts of the two (documents and their datapack candidates, +accessors, skins, the animation samplers) can move into Kazarov as one layer later, once both games +are ready for it. + +**Skinning** stays on the CPU, as today. Kazarov's GPU skinning path can come later, measured. + +--- + +## 6. Datapacks + +**Later.** For now the loader takes a bundle only where it rebuilds the disc's bytes (section 12, +"Safeguards"), so a datapack's model isn't loaded yet. The plan: + +A datapack replaces a model by its published path with its own glTF (and optionally its sidecar). +A replacement may change topology, materials and textures; its actions are matched by name, and a +replacement that lacks one of the original's actions, or is malformed, falls back to the published +model, as ChronicleTwo's loader does. + +--- + +## 7. What lives where + +| Kazarov (shared) | Chronicle | +|---|---| +| MDS/MDT/MOT/WGT/BBP/CHR to glTF; textures linked | the pack layout and which files are models | +| the character configuration as typed commands, from a game's table | Dark Cloud's command table and `.cfg` reading | +| the motion name table | Dark Cloud's names added to it | +| | the glTF loader, which rebuilds the retail records from the bundles | +| | `CCloth`'s step, original and enhanced (section 8), and the Options row | + +--- + +## 8. Cloth + +**What it is.** A `.clo` file sets up one piece of cloth: `SIZE` (a grid of up to 16 by 16 points), +`FRAME` (the object it hangs from), `GRAVITY`, `WINDEFFECT`, `FOLLOW` (how much of the body's +movement it copies, per axis), `K` (the stiffness of the springs pulling each point to its rest +shape), `BOUND` (a box on a named object it can't pass, with friction), `NORMAL` and `POLYDIV`. The +grid itself is a mesh of the model. `CCloth` steps it once per tick: each point springs towards its +rest place on its frame, keeps its rest distances to its neighbours, falls under gravity, takes the +wind, and is pushed out of the boxes and off the floor. Toan's turban and poncho are three of them. + +**Converted.** The grid travels in the glTF as a mesh. Its parameters are published as typed +commands in the character's sidecar, as Dark Cloud 2's cloth commands are; the game still reads the +`.clo` itself, as the PS2 does. + +**Original.** The PS2's simulation, as the port runs it now: the cloth looks and moves exactly as it +does on the PS2. + +**Enhanced,** the PS2's step with the body added. After the step pushes the cloth out of the authored +boxes, it pushes it out of a capsule along each joint of the arms and legs (a chain of two joints, +`jntN_1` to `jntN_2` to `effN`), a fifth of the joint's length thick, the same way, so the push carries +into the speeds as a box's does. Two rules keep it from doing harm: + +- A vertex comes no nearer a limb than it rested when the cloth settled on the body (first step, or + after a cut): cloth authored lying against an arm isn't pushed off it, and an arm raised through + where the cloth rests still pushes it. Skipping a limb wherever the rest place lay inside it let + Toan's arms pass through the poncho's shoulders when he raises them over the snake in the attract + movie: overlaps up to 0.74 deep, now at most 0.34 there. +- A limb pressing a vertex towards the torso (a one-joint chain whose end the other chains branch from: + the pelvis, the spine up to the shoulders) stops it at the torso's surface, or where it rests if that + is nearer. An arm swinging in front of Toan's poncho pressed it into his chest, + which the PS2's cloth never shows; squeezed between the two, the cloth now stays on the chest and the + arm overlaps it. The push is `cloth_body::PushOutOfBody` (`port/include/game/cloth.hpp`), tested on its own. + +The torso's surface is the body's own mesh. When the cloth settles, the skinned meshes the character +draws (Toan's `skin`; a cloth's frame is not drawn, so the poncho's starting mesh is left out) are read +as they stand, each vertex given to the bone whose capsule surface it lies nearest (the head, hands and +feet among them, so a hand at his side doesn't widen him), and the torso's faces are measured from the +spine outwards in 12 directions at 6 heights, in the spine joint's frame so the shape turns with him. +The steps then only look the shape up. A round torso 0.4 of the spine's length thick, the first +version, reached 1.29 all round: right for Toan's chest (1.26 at its deepest) but 0.3 to 0.7 out behind +his back (0.6 at the waist, 1.0 at the shoulders), so it held the poncho's back panel off him (a +quarter of the time, by up to 0.4) where the PS2 lets it lie. Fitted, the back hangs as the PS2's does +(its rows' mean distances from the spine within 0.01) and never goes into his back, and the front +stays off the chest as well as before. A torso the mesh says too little about (fewer than 8 vertices, +or faces in under half the directions: Toan's pelvis, whose vertices the legs take) stays round. + +The torso also keeps the cloth off the chest: out to its surface, or for cloth authored resting inside +it (Toan's collar, up to 0.8 deep in his chest mesh) no deeper than it rests. Pushed to the surface, the +collar puffed; left alone, it sank up to 1.0 deeper than it rests in the attract movie's poses. What +still shows there, the poncho inside his chest and shoulders while he rides Dran, is the PS2's own: the +collar follows its rest place into the chest as his skinned chest and raised shoulders swell 0.3 and +more past their standing shape, Original clips the same, and neither a torso refitted every step nor a +limit of 0.3 under the surface changed it. The PS2's +step carries the cloth along with a share of the body's movement (`FOLLOW`) but none of its turning, +so a quick about-turn (15 degrees a tick, a dozen ticks) swung Toan's chest through the poncho. Carrying +the cloth round with turns faster than 3 degrees a tick, as current engines limit the angular inertia a +cloth feels, was tried first: it stopped the clipping but cut the poncho's movement against the body by +60% running in circles, and it looked stiff. The backstop keeps all of the PS2's movement (0.225 against +0.223 running in circles) and cut cloth inside the chest by 75 to 90%; on an about-turn the poncho still +swings round his side, round the chest instead of through it. + +Everything else is the PS2's, so the cloth looks and moves as it always has, except that arms and legs +no longer pass through it. + +A full position-based (XPBD) solver was tried first and dropped. Dark Cloud's cloth is tuned to the +PS2's soft, four-pass springs: they let the poncho stretch a quarter or more round the authored boxes, +many of whose rest places lie inside them. Held to its lengths, the poncho rode up off the body and +showed the belt; softened enough to hang right, it stretched off the neck, jittered at two and a half +times the PS2's rate and flipped up over Toan's face. + +Both keep the vertices, speeds and normals in `CCloth`, so drawing is unchanged and the two can be +switched at any time. The Options screen's Display page has the Cloth row (Original, Enhanced); +Original is the default until both have been compared in play. + +--- + +## 9. Phases + +All four phases land in one pull request: nothing switches to glTF until everything does, so no +model, cloth included, ever draws without the behaviour it has today. The phases are the order the +work is done in and the gates it passes; each gate's tests (section 10) pass before the next phase +starts. + +1. **Extraction.** Kazarov's model bridge and its configuration hook; Dark Cloud's command table and + actions with their frames; the names added to the shared table; every model published as glTF, + checked against the retail loaders on the disc. +2. **Characters from glTF.** The character, motion and model loaders build from the published glTF; + every pose at every key frame equals retail's. +3. **The rest.** Static scenes, items, weapons, the title and opening, collision models and georama + parts. +4. **Cloth.** The cloth documents; the original step from them; the enhanced mode and its Options + row. + +Later, measured, and separately: GPU skinning, glTF replacements that add actions or bones, and +unifying the glTF loader with ChronicleTwo's. + +--- + +## 10. Tests + +- The converter: Dark Cloud's `KEY` lines become named actions with their frames; the three-file + `MOTION` form gives a skin; `TEX_ANIME` and `CLOTH` reach the sidecar; fixtures made in code. +- Parity on the disc (`DC_MODEL_DATA`, `DC_MODEL_RAW`): every model, weight and bind pose rebuilt + from its bundle equals the disc's bytes; every character motion's channels, sampled at every frame + of every action, match the disc's within 10⁻³, and no motion comes back with more keys than the + disc's (the game sizes its arenas for them). +- The configuration reader: names from comments, missing commas, index-only comments, texture + animations in order, cloth across lines, the game's argument checks. +- Loading: a character from glTF draws the same pose as from its pack at a set of frames; hit-window + frames land on the same poses; `SearchFrame` finds every attachment point. +- Datapacks: a replacement glTF wins; a malformed one, or one missing an action, falls back. +- Cloth: the original step from the document matches today's `CCloth` step for step; the enhanced + one keeps the cloth out of the arms and legs and otherwise moves as the original does. + +--- + +## 11. Decided, and still open + +Decided (r2): + +- **Motion names** are translated into English for what the motion or model is for, and reviewed. +- **The glTF loader is Chronicle's own;** it is unified with ChronicleTwo's later. +- **Every model** uses the new format, georama parts and collision models included. +- **One pull request** for phases 1 to 4, which serve as test gates. + +Decided (r3): + +- **Kazarov takes a game profile;** the configuration syntax stays in Chronicle. +- **The game's loaders stay;** Chronicle rebuilds their records from the bundles. + +Open: + +- **The cloth default,** once Original and Enhanced can be compared. +- **Size.** Converted models take 1.9 GB before packing, against 650 MB on the disc, mostly from + motion packs placed into every model they fit (as for Dark Cloud 2). Each could instead live once, + for Dark Cloud's motions apply by frame index. +- **Stripping by default.** `extract` keeps the disc's files beside the glTF for now; stripping is + a separate step until loading from glTF is proven in play. + +--- + +## 12. Status + +- **Phase 1, extraction:** done. `extract` converts the models with the textures; + `dcdata convert-models ` converts the disc into a folder of its own. +- **Phase 2, loading:** on the PAL disc, every model, weight and bind pose rebuilds byte for byte, + and every one of the 2,450 motions keeps its poses in every action (largest difference 9×10⁻⁶), + never with more keys than the disc's. The loader rebuilds every source as its file is read. + `dcdata bake-assets ` publishes the models into a data folder; baked again over its + own output, it leaves out what it published before (`.assets` folders, glTF, buffers, sidecars, + `models.json`), which it would otherwise publish again beside the new files. Tested in play + in Norune and the attract movie. +- **Phase 3, the rest:** every model source on the disc is indexed (11,949: scenes, georama `.scn` + slots, dungeon `.mpd` parts, the opening's `.sne`, title and menu packs, collision models, items, + weapons, cutscene motion sets). `dcdata strip-models ` empties the converted sources in + place; stripped data reads back as the disc's. +- **Phase 4, cloth:** Enhanced (the PS2's step plus the body) and its Options row. Tests: on a + skeleton, Enhanced keeps the cloth out of a leg the Original passes through; an arm pushes cloth out + of it, presses it only as far as the chest and never deeper than it rests, and leaves cloth that + rests against it alone; the chest keeps cloth off itself, and cloth authored inside it no deeper than + it rests; the torso fits a body's mesh, deep chest and shallow back, without a hand at its side + widening it. In headless runs of Toan running, turning and stopping in Norune, Enhanced moves + as the Original does (same tick-to-tick motion, neck stretch and hem) while the limbs push the + poncho hundreds of times; through quick about-turns and running in circles the poncho swings as on + the PS2 but round his chest rather than through it. +- **Later:** datapack models (section 6), the game reading cloth from its typed commands, GPU + skinning, and unifying the loader with ChronicleTwo's. + +**Safeguards.** Each source's size and CRC-32 are in `models.json`. The loader writes a rebuilt +source only over the bytes it was converted from, or the zeros stripping left of them, so a pack a +mod changed keeps what it has. A rebuilt model, weight or bind pose must match its CRC, or the file +keeps what it read. A motion keeps the disc's key frames in every channel: the game reads every +vertex channel of a frame at the first one's key, and blends towards keys by frame. The channels' +order matters for the same reason: rebuilt sorted by frame, Saget's vertex animation in the attract +movie, read against the wrong first channel, wrote past his mesh and crashed its draw. So the sidecar +lists every channel of a motion in its order, with its header words (the same three in every channel +of a file: tool leftovers), and keeps with their words the keys a clip can't give back bit for bit +(glTF's unit rotations change the last bit of a disc rotation that wasn't quite one); every motion +rebuilds as the disc has it, and one that doesn't goes into a pack only if it matches the disc's CRC. +`ModelGltf.MotionsGoInOnlyAsTheDiscHasThem` checks every motion in every pack. Stripped data +(`models.stripped`) refuses to run, with a message to rebake, when its index is missing or a +source can't be rebuilt, rather than handing the game zeros. diff --git a/docs/PC.md b/docs/PC.md index c800962cc..ab40a5041 100644 --- a/docs/PC.md +++ b/docs/PC.md @@ -176,7 +176,8 @@ key is optional; these are the defaults: "detail_distance": 0, // how far full detail reaches (world units); 0: at any distance "shadow_distance": 0, // how far town parts cast full shadows (world units); 0: at any distance "anisotropy": 0, // anisotropic filtering of 3D textures: 0 (off), 2, 4, 8 or 16 samples - "soft_focus": true // the game's farside soft focus + "soft_focus": true, // the game's farside soft focus + "cloth": "original" // original: the PS2's cloth; enhanced: also kept out of the body (docs/MODEL_FRAMEWORK.md) }, "audio": { "master_volume": 1.0, // 0 to 1 @@ -1394,7 +1395,7 @@ help. Retail's screen-position row is gone: `MGAdjustScreen` moves nothing on PC | Page | Rows | |---|---| | Game | save cursor position, clock, time speed, dungeon map, enemy damage, party damage, enemy HP, names, Discord Rich Presence, element quick select, swap keeps lock-on, autosave | -| Display | window mode, resolution (the monitor's own and the sizes that fit it), V-Sync (`fifo`, `mailbox`, `immediate`), frame limit, aspect ratio, interface size (`ui_scale`), smooth motion (`interpolation`), FPS counter, FPS info (`fps_detail`), anisotropic filter (`anisotropy`), soft focus | +| Display | window mode, resolution (the monitor's own and the sizes that fit it), V-Sync (`fifo`, `mailbox`, `immediate`), frame limit, aspect ratio, interface size (`ui_scale`), smooth motion (`interpolation`), FPS counter, FPS info (`fps_detail`), anisotropic filter (`anisotropy`), soft focus, cloth (`cloth`) | | Audio | volume, sound (mono, stereo or surround), soundtrack (PS2 or custom) | | Controls | vibration, mouse sensitivity (in hundredths below 1 and tenths above, whatever its unit), invert mouse Y, vertical return, mouse wheel zoom, reset zoom (its binding), stick sensitivity, invert stick X and Y, gyro, gyro sensitivity, invert gyro X and Y | | Text | language, message speed, button symbols (original or new), symbols shown (glyph device), text font (sharp or original), text shadow, symbol shadow, area name shadow, floor label shadow, boss name shadow | @@ -1952,6 +1953,12 @@ first. `list` prints each file's sector, size and path. `publish-sound` converts an extracted folder's sound packs into ChronicleTwo's bank, sample and sequence formats beside the originals, which the game still plays (`docs/AUDIO_FRAMEWORK.md`). +`extract` also converts the models, motions, skin weights and bind poses to +glTF beside the originals, indexed by `models.json`; the game rebuilds the +retail records from them as it reads a file. `convert-models ` +writes that conversion to a folder of its own, `bake-assets ` +publishes it into extracted data, and `strip-models ` empties the +converted originals (`docs/MODEL_FRAMEWORK.md`). The layout of `data/` or the root of `data.zip` is the disc archive's own: `dun/pack/maindat.pac`, `commenu/a_eng/savetex.pak`, `sound/bgm/...` and so on. Lookups fold case, diff --git a/port/CMakeLists.txt b/port/CMakeLists.txt index 11c4021fb..00dbf6cec 100644 --- a/port/CMakeLists.txt +++ b/port/CMakeLists.txt @@ -167,7 +167,7 @@ add_library(dc_port OBJECT ${PORT_LIB_SOURCES} ${DC_SCE_SOURCES}) if(NINTENDO_SWITCH) dc_switch_port(dc_port) endif() -target_link_libraries(dc_port PRIVATE dc_options SDL3::SDL3 Vulkan::Vulkan nlohmann_json::nlohmann_json kz_port_headers) +target_link_libraries(dc_port PRIVATE dc_options SDL3::SDL3 Vulkan::Vulkan nlohmann_json::nlohmann_json fastgltf::fastgltf kz_port_headers) # port/src, the generated headers and the stb directories: captionfont.cpp rasterizes with # stb_truetype and platform/png_image.cpp decodes with stb_image, which Kazarov stages (or fetches) # for dc_host. diff --git a/port/include/game/cloth.hpp b/port/include/game/cloth.hpp index 282566e5a..65627b88f 100644 --- a/port/include/game/cloth.hpp +++ b/port/include/game/cloth.hpp @@ -4,6 +4,11 @@ #include +#include +#include +#include +#include + #include "dataalloc.hpp" #include "mdt.hpp" #include "visualvu1.hpp" @@ -104,4 +109,70 @@ class CCloth : public CVisualVu1 { virtual int DrawVu1(sceVif1Packet *packet, float (*matrix)[4], RenderInfo *info, VU1_PROGRAM program, u_long128 *draw_state, int unknown1, int unknown2); }; +/** + * The Options screen's Enhanced cloth pushes the cloth out of the body as well as the .clo files' + * boxes. The body is the skeleton as capsules. + */ +namespace cloth_body { + +constexpr int kTorsoSectors = 12; /**< How many directions round its axis a torso's shape is measured in. */ +constexpr int kTorsoBands = 6; /**< How many heights along its axis a torso's shape is measured at. */ + +/** + * A capsule of the body, from a to b, radius thick. + */ +struct Capsule { + std::array a; + std::array b; + float radius; + bool torso = false; /**< Keeps the cloth off itself, and stops the limbs pressing the cloth into it. */ + /** + * A torso fitted to the body's mesh (FitTorsos): how far its surface lies from the axis, by direction + * round it (measured from `side`, a direction square to the axis that turns with the body) and by + * height from a to b. Unfitted, the torso is round, `radius` thick. + */ + std::array side = {}; + std::array shape = {}; + bool fitted = false; +}; + +/** + * How far the capsule's surface lies from its axis in the direction of `p`. + */ +float Reach(const Capsule &capsule, const float *p); + +/** + * The body's mesh: its vertices, in world space, and its triangles, three vertices each. + */ +struct Mesh { + std::vector> vertices; + std::vector> triangles; +}; + +/** + * Fits the torsos among `capsules` to `body` as it stands: how far out from the axis each direction meets + * the torso's faces. A vertex belongs to the capsule whose surface it lies nearest, `others` (the head, + * the hands, the feet) among them, so arms and hands held against the torso don't fatten it. A round + * capsule sized from the spine's length stood out behind Toan's back, his chest being deeper than it. + */ +void FitTorsos(std::span capsules, std::span others, const Mesh &body); + +/** + * How near a limb's axis a vertex may come: the limb's thickness, or less where the cloth rests against + * the limb, as near as `home` (where the cloth's rest shape puts the vertex when it settles on the body) + * lies to it then. A vertex resting against an arm isn't pushed off it, and an arm raised through where + * the cloth rests still pushes it. + */ +float Clearance(const Capsule &limb, const float *home); + +/** + * Pushes a free vertex out of `capsules`, each limb to its `clearances` entry (Clearance, taken when the + * cloth settled), the torso to its surface, or as far out as `home` lies if that is inside it. A limb + * pressing the vertex towards the torso stops it at the torso's surface, or where it rests if that is + * nearer. + */ +void PushOutOfBody(std::span capsules, std::span clearances, const float *home, float *vertex); + +} // namespace cloth_body + } // namespace dc::game diff --git a/port/lang/de_de.json b/port/lang/de_de.json index 5db037b04..693c0b457 100644 --- a/port/lang/de_de.json +++ b/port/lang/de_de.json @@ -91,6 +91,10 @@ "options.video.soft_focus.label": "Weichzeichner", "options.video.soft_focus.choice.0": "An", "options.video.soft_focus.choice.1": "Aus", + "options.video.cloth.label": "Stoff", + "options.video.cloth.help": "\"Stoff\"\nOriginal: wie auf der PS2.\nVerbessert: der Körper\ngeht nicht mehr hindurch.", + "options.video.cloth.choice.0": "Original", + "options.video.cloth.choice.1": "Verbessert", "options.page.audio": "Audio", "options.audio.master_volume.label": "Lautstärke", "options.audio.master_volume.help": "\"Lautstärke\"\nWie laut das Spiel ist.", diff --git a/port/lang/es_es.json b/port/lang/es_es.json index 4ab994458..ec974d09f 100644 --- a/port/lang/es_es.json +++ b/port/lang/es_es.json @@ -91,6 +91,10 @@ "options.video.soft_focus.label": "Enfoque suave", "options.video.soft_focus.choice.0": "Sí", "options.video.soft_focus.choice.1": "No", + "options.video.cloth.label": "Tela", + "options.video.cloth.help": "\"Tela\"\nOriginal: la de PS2.\nMejorada: el cuerpo\nya no la atraviesa.", + "options.video.cloth.choice.0": "Original", + "options.video.cloth.choice.1": "Mejorada", "options.page.audio": "Audio", "options.audio.master_volume.label": "Volumen", "options.audio.master_volume.help": "\"Volumen\"\nQué tan alto suena.", diff --git a/port/lang/fr_fr.json b/port/lang/fr_fr.json index 3e3e57be0..3596beb85 100644 --- a/port/lang/fr_fr.json +++ b/port/lang/fr_fr.json @@ -91,6 +91,10 @@ "options.video.soft_focus.label": "Flou doux", "options.video.soft_focus.choice.0": "Activé", "options.video.soft_focus.choice.1": "Désactivé", + "options.video.cloth.label": "Tissus", + "options.video.cloth.help": "\"Tissus\"\nOriginal: celui de la PS2.\nAmélioré: le corps\nne le traverse plus.", + "options.video.cloth.choice.0": "Original", + "options.video.cloth.choice.1": "Amélioré", "options.page.audio": "Audio", "options.audio.master_volume.label": "Volume", "options.audio.master_volume.help": "\"Volume\"\nLe volume du jeu.", diff --git a/port/lang/it_it.json b/port/lang/it_it.json index 3bb1360fb..a914df226 100644 --- a/port/lang/it_it.json +++ b/port/lang/it_it.json @@ -91,6 +91,10 @@ "options.video.soft_focus.label": "Fuoco morbido", "options.video.soft_focus.choice.0": "Sì", "options.video.soft_focus.choice.1": "No", + "options.video.cloth.label": "Tessuti", + "options.video.cloth.help": "\"Tessuti\"\nOriginale: come su PS2.\nMigliorato: il corpo\nnon li attraversa più.", + "options.video.cloth.choice.0": "Originale", + "options.video.cloth.choice.1": "Migliorato", "options.page.audio": "Audio", "options.audio.master_volume.label": "Volume", "options.audio.master_volume.help": "\"Volume\"\nQuanto forte è il\ngioco.", diff --git a/port/src/data/data_read.cpp b/port/src/data/data_read.cpp index d193c13f8..6a513f3b1 100644 --- a/port/src/data/data_read.cpp +++ b/port/src/data/data_read.cpp @@ -25,6 +25,7 @@ #include #include "../../../tools/dcdata/dcdata.hpp" +#include "data/model_gltf.hpp" #include "dataalloc.hpp" #include "platform/data_archive.hpp" #include "platform/load_work_profile.hpp" @@ -109,6 +110,7 @@ std::unordered_map loaded_capaci std::unordered_map> copied_capacities_by_owner; constinit std::atomic copied_capacity_count = 0; bool data_indexed; +constinit std::atomic data_generation = 0; std::mutex texture_read_stats_mutex; DataTextureReadStats texture_read_stats; std::atomic texture_read_stats_enabled = false; @@ -375,6 +377,7 @@ std::string IndexedPath(const IndexedFile &file) { } // namespace void DataReadIndex() { + data_generation.fetch_add(1, std::memory_order_relaxed); ResetMetadataSnapshots(); const fs::path &root = PathsDataRoot(); std::error_code error; @@ -502,16 +505,25 @@ bool DataReadLoadFile(std::string_view path, void *buffer, int *size) { ForgetPackCapacities(buffer); std::size_t capacity = 0; *size = ReadWhole(*file, buffer, &capacity); + model_gltf::Splice(IndexedPath(*file), std::span(static_cast(buffer), static_cast(*size))); ForgetOverwrittenCapacities(buffer, capacity); TrackPackCapacity(buffer, capacity, buffer, IndexedPath(*file), {}, 0, static_cast(*size)); return true; } +bool DataFileExists(std::string_view path) { + return Lookup(path) != nullptr; +} + std::uintmax_t DataFileSize(std::string_view path) { const IndexedFile *file = Lookup(path); return file != nullptr ? file->size : 0; } +std::uint64_t DataReadGeneration() { + return data_generation.load(std::memory_order_relaxed); +} + bool DataReadIndexed() { return data_indexed; } @@ -757,6 +769,7 @@ void DataReadForgetPackCapacities(const void *owner) { } void DataReadReset() { + model_gltf::Reset(); ResetMetadataSnapshots(); loaded_capacities.clear(); source_index.clear(); diff --git a/port/src/data/data_read.hpp b/port/src/data/data_read.hpp index 24f9bac76..384cc5b32 100644 --- a/port/src/data/data_read.hpp +++ b/port/src/data/data_read.hpp @@ -29,6 +29,16 @@ std::vector SupportedLanguages(); */ std::uintmax_t DataFileSize(std::string_view path); +/** + * Whether the game's data has a file of that name, even an empty one. + */ +bool DataFileExists(std::string_view path); + +/** + * How many times the data has been indexed: what read it before can be stale once this changes. + */ +std::uint64_t DataReadGeneration(); + /** * Whether the data has been indexed. Unlike the readers it never indexes, which stops the program * when there is no data. diff --git a/port/src/data/model_gltf.cpp b/port/src/data/model_gltf.cpp new file mode 100644 index 000000000..0056b7f4e --- /dev/null +++ b/port/src/data/model_gltf.cpp @@ -0,0 +1,1027 @@ +#include "data/model_gltf.hpp" + +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../../../tools/dcdata/dcdata.hpp" +#include "data/data_read.hpp" + +namespace dc::data::model_gltf { +namespace { + +using Json = nlohmann::json; + +// A published bundle: the glTF with its buffers attached, and the sidecar beside it. +struct Bundle { + fastgltf::Asset asset; + Json sidecar; +}; + +struct State { + std::mutex mutex; + bool indexed = false; + std::uint64_t generation = 0; + std::map index; + std::unordered_map> bundles; + std::deque recent; + std::set failed; + bool stripped = false; + std::unordered_map>> rebuilt; + std::deque rebuilt_order; + std::size_t rebuilt_bytes = 0; +}; + +// Rebuilt sources are kept, up to this many bytes, for the files the game reads again. +constexpr std::size_t kKeptRebuiltBytes = 128u << 20; + +State &Shared() { + static State state; + return state; +} + +// Kazarov names sources as the disc spells them; the port's loaders fold paths as the game does. +std::string Key(std::string_view alias) { + return dcdata::FoldPath(alias); +} + +void LoadIndex(State &state); + +// Reads the index, again when the data has been indexed again since. +void Refresh(State &state) { + // Reading the index may index the data, so the generation is taken after. + if (!state.indexed || state.generation != DataReadGeneration()) { + LoadIndex(state); + state.generation = DataReadGeneration(); + } +} + +// Forgets everything read from the data, with the state locked. +void Forget(State &state) { + state.indexed = false; + state.index.clear(); + state.bundles.clear(); + state.recent.clear(); + state.failed.clear(); + state.stripped = false; + state.rebuilt.clear(); + state.rebuilt_order.clear(); + state.rebuilt_bytes = 0; +} + +// Stops the game: data whose sources were emptied can't run without the models that rebuild them. +[[noreturn]] void Unrecoverable(const std::string &why) { + std::fprintf(stderr, + "%s\nThe game data's models were stripped (dcdata strip-models), so the game can't load them " + "from the disc's files either. Extract the data again, or run dcdata bake-assets on it.\n", + why.c_str()); + std::fflush(stderr); + std::abort(); +} + +void LoadIndex(State &state) { + // Indexing the data reads it anew; what was read from it before may be stale. + Forget(state); + state.indexed = true; + state.stripped = DataFileExists("models.stripped"); + std::vector bytes; + if (!DataReadReadFile("models.json", bytes)) { + if (state.stripped) { + Unrecoverable("models.json is missing"); + } + return; + } + const Json index = Json::parse(bytes.begin(), bytes.end(), nullptr, false); + if (!index.is_object()) { + if (state.stripped) { + Unrecoverable("models.json can't be read"); + } + std::fprintf(stderr, "models.json: not a source index\n"); + return; + } + for (const auto &[alias, entry] : index.items()) { + state.index.emplace(Key(alias), entry); + } +} + +// Where a bundle's file sits in the data (dcdata::PublishedPath). +std::string OnDisk(std::string_view path) { + return dcdata::PublishedPath(path, [](std::string_view prefix) { return DataFileExists(prefix); }); +} + +std::string Directory(const std::string &path) { + const std::size_t slash = path.rfind('/'); + return slash == std::string::npos ? std::string() : path.substr(0, slash + 1); +} + +// Bundles are parsed once and kept while they are among the last used: a character's model, +// motions and weights come from one bundle, a placed motion's from two, and a scene file holds many. +constexpr std::size_t kKeptBundles = 16; + +std::shared_ptr LoadBundle(State &state, const std::string &path) { + if (auto found = state.bundles.find(path); found != state.bundles.end()) { + std::erase(state.recent, path); + state.recent.push_back(path); + return found->second; + } + std::vector gltf; + std::vector sidecar; + const std::string stem = path.substr(0, path.size() - std::string_view(".gltf").size()); + if (!DataReadReadFile(OnDisk(path), gltf) || !DataReadReadFile(OnDisk(stem + ".game.json"), sidecar)) { + throw std::runtime_error("missing bundle " + path); + } + auto input = fastgltf::GltfDataBuffer::FromBytes(reinterpret_cast(gltf.data()), gltf.size()); + if (input.error() != fastgltf::Error::None) { + throw std::runtime_error("cannot read " + path); + } + fastgltf::Parser parser; + // Buffers are read through the data's own reader below, archived or loose alike. + auto parsed = parser.loadGltf(input.get(), {}, fastgltf::Options::None); + if (parsed.error() != fastgltf::Error::None) { + throw std::runtime_error(path + ": " + std::string(fastgltf::getErrorMessage(parsed.error()))); + } + if (fastgltf::validate(parsed.get()) != fastgltf::Error::None) { + throw std::runtime_error(path + ": invalid glTF"); + } + auto bundle = std::make_shared(Bundle{std::move(parsed.get()), Json::parse(sidecar.begin(), sidecar.end())}); + for (fastgltf::Buffer &buffer : bundle->asset.buffers) { + const auto *uri = std::get_if(&buffer.data); + if (uri == nullptr) { + continue; + } + std::vector data; + if (!DataReadReadFile(OnDisk(Directory(path) + std::string(uri->uri.path())), data) || + data.size() < buffer.byteLength) { + throw std::runtime_error("missing buffer of " + path); + } + fastgltf::sources::Vector storage; + storage.bytes.resize(data.size()); + std::memcpy(storage.bytes.data(), data.data(), data.size()); + buffer.data = std::move(storage); + } + // Holders keep their bundles alive past an eviction. + while (state.recent.size() >= kKeptBundles) { + state.bundles.erase(state.recent.front()); + state.recent.pop_front(); + } + state.recent.push_back(path); + state.bundles[path] = bundle; + return bundle; +} + +// A retail record being written: words and floats at the offsets the source file had them. +class Record { +public: + void Size(std::size_t size) { + if (bytes_.size() < size) { + bytes_.resize(size); + } + } + + void Word(std::size_t at, std::uint32_t value) { Put(at, &value, 4); } + + void Float(std::size_t at, float value) { Put(at, &value, 4); } + + void Bytes(std::size_t at, std::string_view value) { Put(at, value.data(), value.size()); } + + std::vector Take() { return std::move(bytes_); } + +private: + void Put(std::size_t at, const void *value, std::size_t size) { + Size(at + size); + std::memcpy(bytes_.data() + at, value, size); + } + + std::vector bytes_; +}; + +// The CRC-32 (IEEE) of some bytes, as models.json records each source's. +std::uint32_t Crc32(std::span bytes) { + static const std::array table = [] { + std::array out{}; + for (std::uint32_t n = 0; n < 256; n++) { + std::uint32_t c = n; + for (int k = 0; k < 8; k++) { + c = c & 1 ? 0xEDB88320u ^ (c >> 1) : c >> 1; + } + out[n] = c; + } + return out; + }(); + std::uint32_t crc = 0xFFFFFFFFu; + for (std::uint8_t b : bytes) { + crc = table[(crc ^ b) & 0xFF] ^ (crc >> 8); + } + return crc ^ 0xFFFFFFFFu; +} + +std::uint32_t U32(const Json &value) { + return value.get(); +} + +void Padding(Record &record, std::size_t base, const Json &holder) { + for (const Json &word : holder.value("padding_u32", Json::array())) { + record.Word(base + U32(word[0]), U32(word[1])); + } +} + +// The values of one retail stream, by source index, as the glTF primitives' corners carry them. +using Stream = std::map>; + +template +void Gather(const fastgltf::Asset &asset, std::size_t accessor, const Json &sources, Stream &stream) { + fastgltf::iterateAccessorWithIndex(asset, asset.accessors[accessor], [&](Vector value, std::size_t corner) { + std::array &entry = stream[U32(sources.at(corner))]; + for (std::size_t component = 0; component < Width; component++) { + entry[component] = value[component]; + } + }); +} + +// Rebuilds one MDT at `base`: its header and mesh-section words from the sidecar, its streams from +// the glTF corners and the sidecar's overrides, its strips from the corners' source indices. +void Mesh(Record &record, std::size_t base, const Bundle &bundle, const Json &mesh, const fastgltf::Node *node) { + const Json &words = mesh.at("record_u32"); + for (std::size_t k = 0; k < 16; k++) { + record.Word(base + k * 4, U32(words[k])); + } + const std::size_t face = U32(words[10]); + for (std::size_t k = 0; k < 4; k++) { + record.Word(base + face + k * 4, U32(words[16 + k])); + } + record.Size(base + U32(words[2])); + auto signed_word = [&](std::size_t k) { return static_cast(U32(words[k])); }; + const std::array counts = {U32(words[3]), U32(words[5]), U32(words[11]), U32(words[7])}; + const std::array offsets = {signed_word(4), signed_word(6), signed_word(12), signed_word(8)}; + + std::array streams; + const bool collision = mesh.value("kind", "") == "collision"; + if (!collision && node != nullptr && node->meshIndex) { + const fastgltf::Mesh &gltf_mesh = bundle.asset.meshes[*node->meshIndex]; + const Json &primitives = mesh.at("primitives"); + for (std::size_t p = 0; p < gltf_mesh.primitives.size() && p < primitives.size(); p++) { + const fastgltf::Primitive &primitive = gltf_mesh.primitives[p]; + const Json &sources = primitives[p]; + if (auto *a = primitive.findAttribute("POSITION"); a != primitive.attributes.end()) { + Gather(bundle.asset, a->accessorIndex, sources.at("source_vertices"), streams[0]); + } + if (auto *a = primitive.findAttribute("NORMAL"); a != primitive.attributes.end()) { + Gather(bundle.asset, a->accessorIndex, sources.at("source_normals"), streams[1]); + } + if (auto *a = primitive.findAttribute("TEXCOORD_0"); a != primitive.attributes.end()) { + Gather(bundle.asset, a->accessorIndex, sources.at("source_uvs"), streams[2]); + } + if (auto *a = primitive.findAttribute("COLOR_0"); a != primitive.attributes.end()) { + Gather(bundle.asset, a->accessorIndex, sources.at("source_colours"), streams[3]); + } + } + } + // A component the sidecar doesn't override is the glTF's, or the standard default: 1 for a + // position's w, 0 elsewhere. + constexpr std::array kNames = {"positions", "normals", "uvs", "colours"}; + const Json overrides = mesh.value("stream_overrides", Json::object()); + for (std::size_t s = 0; s < 4; s++) { + if (offsets[s] <= 0) { + continue; + } + for (std::uint32_t i = 0; i < counts[s]; i++) { + std::array value = {0, 0, 0, 0}; + if (auto found = streams[s].find(i); found != streams[s].end()) { + value = found->second; + if (s == 0) { + value[3] = 1; + } + } + for (std::size_t c = 0; c < 4; c++) { + record.Float(base + offsets[s] + i * 16 + c * 4, value[c]); + } + } + for (const Json &entry : overrides.value(kNames[s], Json::array())) { + record.Float(base + offsets[s] + U32(entry[0]) * 16 + U32(entry[1]) * 4, entry[2].get()); + } + } + + const std::size_t info = U32(words[14]); + if (collision) { + const Json &polygons = mesh.at("polygons"); + record.Word(base + face + 16, U32(mesh.at("face_reserved_u32")[2])); + record.Word(base + face + 20, static_cast(polygons.size())); + for (std::size_t k = 0; k < polygons.size(); k++) { + for (std::size_t w = 0; w < 5; w++) { + record.Word(base + face + 24 + k * 20 + w * 4, U32(polygons[k][w])); + } + } + const Json &records = mesh.at("info"); + for (std::size_t k = 0; k < records.size(); k++) { + for (std::size_t w = 0; w < 4; w++) { + record.Word(base + info + k * 16 + w * 4, U32(records[k][w])); + } + } + } else { + // Every corner indexes a position, a normal and a texture coordinate, and a colour when the + // mesh has colours; a strip with no corners keeps its place. + const std::size_t stride = offsets[3] > 0 ? 4 : 3; + std::map empty; + for (const Json &strip : mesh.value("empty_strips", Json::array())) { + empty[U32(strip.at("index"))] = &strip; + } + const Json &primitives = mesh.at("primitives"); + std::size_t at = base + face + 16; + std::size_t next = 0; + for (std::uint32_t s = 0; s < U32(words[18]); s++) { + if (auto found = empty.find(s); found != empty.end()) { + record.Word(at, U32(found->second->at("type"))); + record.Word(at + 4, 0); + record.Word(at + 8, U32(found->second->at("material"))); + at += 12; + continue; + } + const Json &strip = primitives.at(next++); + const Json &v = strip.at("source_vertices"); + record.Word(at, U32(strip.at("type"))); + record.Word(at + 4, static_cast(v.size())); + record.Word(at + 8, U32(strip.at("material"))); + at += 12; + for (std::size_t c = 0; c < v.size(); c++) { + record.Word(at, U32(v[c])); + record.Word(at + 4, U32(strip.at("source_normals")[c])); + record.Word(at + 8, U32(strip.at("source_uvs")[c])); + if (stride == 4) { + record.Word(at + 12, U32(strip.at("source_colours")[c])); + } + at += stride * 4; + } + } + const Json &materials = mesh.at("materials"); + for (std::size_t k = 0; k < materials.size(); k++) { + const Json &material = materials[k].at("record_u32"); + for (std::size_t w = 0; w < 24; w++) { + record.Word(base + info + k * 96 + w * 4, U32(material[w])); + } + } + } + Padding(record, base, mesh); +} + +// Rebuilds an MDS: its header and object records, each object's MDT where it was. +std::vector Scene(const Bundle &bundle, const Json &model) { + Record record; + const Json &words = model.at("record_u32"); + for (std::size_t k = 0; k < 4; k++) { + record.Word(k * 4, U32(words[k])); + } + const std::size_t node_offset = model.value("node_offset", 0u); + std::size_t at = U32(words[3]); + const Json &frames = model.at("frames"); + for (std::size_t f = 0; f < frames.size(); f++) { + const Json &frame = frames[f]; + const std::uint32_t step = U32(frame.at("record_u32")[0]); + const std::uint32_t mdt = U32(frame.at("record_u32")[1]); + const fastgltf::Node &node = bundle.asset.nodes.at(node_offset + f); + record.Word(at, U32(frame.at("reserved_u32")[0])); + record.Word(at + 4, step); + record.Size(at + 40); + record.Bytes(at + 8, std::string_view(node.name).substr(0, 32)); + record.Word(at + 40, mdt); + record.Word(at + 44, static_cast(frame.at("parent").get())); + const Json &matrix = frame.at("local_matrix"); + for (std::size_t k = 0; k < 16; k++) { + record.Float(at + 48 + k * 4, matrix[k].get()); + } + if (!frame.at("mdt").is_null()) { + // A frame whose transform needs a shear helper hangs its mesh on the helper. + const std::size_t mesh_node = frame.value("rotation_node", node_offset + f); + const fastgltf::Node *holder = &bundle.asset.nodes.at(mesh_node); + Mesh(record, mdt, bundle, model.at("meshes").at(frame.at("mdt").get()), holder); + } + at += step; + } + Padding(record, 0, model); + // A scene inside another file spans what its records cover, which can end before its last MDT's + // header says it does. + std::vector bytes = record.Take(); + if (const auto size = model.find("size"); size != model.end()) { + bytes.resize(size->get()); + } + return bytes; +} + +// Rebuilds a channel file (a MOT or a WGT): 32-byte channel records, each followed by its 32-byte +// keys; a channel's sixth word is the distance to the next, 0 on the last. +std::vector Channels(const std::vector &channels) { + Record record; + std::size_t at = 0; + for (std::size_t c = 0; c < channels.size(); c++) { + const Json &channel = *channels[c]; + const Json &keys = channel.at("keys"); + const Json &reserved = channel.at("reserved_u32"); + record.Word(at, static_cast(channel.at("frame").get())); + record.Word(at + 4, static_cast(channel.at("target").get())); + record.Word(at + 8, static_cast(channel.at("type").get())); + record.Word(at + 12, U32(reserved[0])); + record.Word(at + 16, static_cast(keys.size())); + record.Word(at + 20, c + 1 < channels.size() ? static_cast(32 + keys.size() * 32) : 0); + record.Word(at + 24, U32(reserved[1])); + record.Word(at + 28, U32(reserved[2])); + at += 32; + for (const Json &key : keys) { + record.Word(at, U32(key.at("frame"))); + for (std::size_t w = 0; w < 3; w++) { + record.Word(at + 4 + w * 4, U32(key.at("reserved_u32")[w])); + } + // A key whose unused lanes held bits that aren't numbers keeps its words. + if (const auto words = key.find("value_u32"); words != key.end()) { + for (std::size_t w = 0; w < 4; w++) { + record.Word(at + 16 + w * 4, U32((*words)[w])); + } + } else { + for (std::size_t w = 0; w < 4; w++) { + record.Float(at + 16 + w * 4, key.at("value")[w].get()); + } + } + at += 32; + } + } + return record.Take(); +} + +std::vector Matrices(const Json &matrices) { + Record record; + for (std::size_t m = 0; m < matrices.size(); m++) { + for (std::size_t k = 0; k < 16; k++) { + record.Float(m * 64 + k * 4, matrices[m][k].get()); + } + } + return record.Take(); +} + +std::vector Pointers(const Json &list) { + std::vector pointers; + for (const Json &item : list) { + pointers.push_back(&item); + } + return pointers; +} + +// The entry of a standalone sidecar's list that came from `alias`. +const Json &FromSource(const Json &list, std::string_view alias) { + for (const Json &item : list) { + if (Key(item.value("alias", "")) == Key(alias)) { + return item; + } + } + throw std::runtime_error("bundle lacks its source"); +} + +// One retail key being rebuilt: its reserved and value words. +struct RetailKey { + std::array reserved{}; + std::array value{}; +}; + +// One retail channel being rebuilt: its header's reserved words and its keys by source frame. +struct Track { + std::uint32_t frame; + std::uint32_t target; + std::int32_t type; + std::array reserved{}; + std::map keys; +}; + +using TrackId = std::tuple; + +// The channels being rebuilt, and the order the sidecar lists them in, the disc's. +struct Tracks { + std::map by_id; + std::vector order; +}; + +Track &TrackOf(Tracks &tracks, std::uint32_t frame, std::int32_t type, std::uint32_t target) { + Track &track = tracks.by_id[{frame, type, target}]; + track.frame = frame; + track.type = type; + track.target = target; + return track; +} + +std::uint32_t Bits(float value) { + return std::bit_cast(value); +} + +// Adds a channel in its place, with its words and its keys, their frames shifted by `start`. +void AddRetail(Tracks &tracks, const Json &channel, std::uint32_t start) { + const TrackId id = {static_cast(channel.at("frame").get()), channel.at("type").get(), + static_cast(channel.value("target", 0))}; + Track &track = TrackOf(tracks, std::get<0>(id), std::get<1>(id), std::get<2>(id)); + tracks.order.push_back(id); + if (const auto reserved = channel.find("reserved_u32"); reserved != channel.end()) { + for (std::size_t w = 0; w < 3; w++) { + track.reserved[w] = U32((*reserved)[w]); + } + } + for (const Json &key : channel.at("keys")) { + RetailKey retail; + if (const auto reserved = key.find("reserved_u32"); reserved != key.end()) { + for (std::size_t w = 0; w < 3; w++) { + retail.reserved[w] = U32((*reserved)[w]); + } + } + // A key a clip can't give back bit for bit keeps its words. + if (const auto words = key.find("value_u32"); words != key.end()) { + for (std::size_t w = 0; w < 4; w++) { + retail.value[w] = U32((*words)[w]); + } + } else { + for (std::size_t w = 0; w < 4; w++) { + retail.value[w] = Bits(key.at("value")[w].get()); + } + } + track.keys[start + U32(key.at("frame"))] = retail; + } +} + +// Reads one action's transform keys back out of its glTF clip: translations and scales as they +// are, rotations from glTF's normalized x, y, z, w to retail's scalar-first, conjugated form. +// Channels the sidecar keeps whole, by frame and type. +using Whole = std::set>; + +void AddClip(Tracks &tracks, const Bundle &bundle, const Json &model, std::size_t node_offset, std::size_t clip, + std::uint32_t start, std::uint32_t end, const Whole &whole) { + std::map> frames; + const Json &model_frames = model.at("frames"); + for (std::uint32_t f = 0; f < model_frames.size(); f++) { + frames[node_offset + f] = {f, false}; + if (const auto helper = model_frames[f].find("rotation_node"); helper != model_frames[f].end()) { + frames[helper->get()] = {f, true}; + } + } + const fastgltf::Animation &animation = bundle.asset.animations.at(clip); + for (const fastgltf::AnimationChannel &channel : animation.channels) { + if (!channel.nodeIndex) { + continue; + } + const auto found = frames.find(*channel.nodeIndex); + if (found == frames.end()) { + continue; + } + const auto [frame, helper] = found->second; + std::int32_t type; + switch (channel.path) { + case fastgltf::AnimationPath::Rotation: + type = 0; + break; + case fastgltf::AnimationPath::Scale: + // A helper's scale keys only undo its authored scale. + if (helper) { + continue; + } + type = 1; + break; + case fastgltf::AnimationPath::Translation: + type = 2; + break; + default: + continue; + } + // A rebaked frame's rotation and scale clips hold its pose, not its keys; the sidecar keeps those. + if (whole.contains({frame, type}) || (type != 2 && model_frames[frame].value("rebaked", false))) { + continue; + } + const fastgltf::AnimationSampler &sampler = animation.samplers[channel.samplerIndex]; + std::vector times; + fastgltf::iterateAccessor(bundle.asset, bundle.asset.accessors[sampler.inputAccessor], + [&](float time) { times.push_back(time); }); + // The clip gives back the keys inside its action; the sidecar keeps those on its ends, and + // those the clip can't give back bit for bit, which win. + Track &track = TrackOf(tracks, frame, type, 0); + auto inside = [&](std::size_t k) -> RetailKey *{ + const std::uint32_t at = start + static_cast(std::lround(times.at(k) * 30.0f)); + if (at <= start || at >= end || track.keys.contains(at)) { + return nullptr; + } + return &track.keys[at]; + }; + if (type == 0) { + fastgltf::iterateAccessorWithIndex( + bundle.asset, bundle.asset.accessors[sampler.outputAccessor], [&](fastgltf::math::fvec4 q, std::size_t k) { + if (RetailKey *key = inside(k)) { + key->value = {Bits(q[3]), Bits(-q[0]), Bits(-q[1]), Bits(-q[2])}; + } + }); + } else { + fastgltf::iterateAccessorWithIndex( + bundle.asset, bundle.asset.accessors[sampler.outputAccessor], [&](fastgltf::math::fvec3 v, std::size_t k) { + if (RetailKey *key = inside(k)) { + key->value = {Bits(v[0]), Bits(v[1]), Bits(v[2]), Bits(0.0f)}; + } + }); + } + } +} + +// Adds the sidecar's retail keys of a motion, noting the channels it keeps whole. +void AddResidue(Tracks &tracks, Whole &whole, const Json &channels) { + for (const Json &channel : channels) { + AddRetail(tracks, channel, 0); + if (channel.value("complete", false)) { + whole.insert({static_cast(channel.at("frame").get()), + channel.at("type").get()}); + } + } +} + +std::vector Write(Tracks &tracks); + +// Rebuilds a character's MOT from its actions: each action's clip for the transforms, the +// sidecar's channels for everything else, and the retail keys of frames glTF can't give back. +std::vector Motion(State &state, const Bundle &own, const Json &resource) { + Tracks tracks; + Whole whole; + AddResidue(tracks, whole, resource.value("retail_channels", Json::array())); + const Bundle *bundle = &own; + std::shared_ptr target; + const Json *placed = &resource; + // A motion placed into its targets refers to its placements, unless this is one of them. + if (const auto external = resource.find("external"); + external != resource.end() && resource.value("variants", Json::array()).empty()) { + placed = nullptr; + const std::string key = external->get(); + for (const Json &path : resource.value("targets", Json::array())) { + target = LoadBundle(state, path.get()); + const Bundle &candidate = *target; + for (const Json &entry : candidate.sidecar.at("resources").at("animations")) { + // A motion placed into models of its own bundle sits beside itself there. + if (entry.value("external", "") == key && + !entry.value("variants", Json::array()).empty()) { + bundle = &candidate; + placed = &entry; + break; + } + } + if (placed != nullptr) { + break; + } + } + } + if (placed != nullptr && !placed->value("variants", Json::array()).empty()) { + const Json &variant = placed->at("variants")[0]; + const Json &model = bundle->sidecar.at("models").at(variant.at("model").get()); + const std::size_t node_offset = variant.at("node_offset").get(); + const Json &actions = placed->at("actions"); + const Json &clips = variant.at("animations"); + for (std::size_t a = 0; a < actions.size(); a++) { + const Json &action = actions[a]; + const std::uint32_t start = U32(action.at("start")); + if (a < clips.size() && !clips[a].is_null()) { + AddClip(tracks, *bundle, model, node_offset, clips[a].get(), start, U32(action.at("end")), + whole); + } + } + } + return Write(tracks); +} + +// Writes rebuilt tracks as a MOT. Its channels keep the disc's key frames, and its order, which the +// sidecar lists: the game reads every vertex channel of a frame at the key of the motion's first +// channel, and blends towards keys by frame. Saget's vertex animation in the attract movie, rebuilt in +// another order, wrote past his mesh and crashed its draw. +std::vector Write(Tracks &tracks) { + std::vector channels; + std::set listed; + for (const TrackId &id : tracks.order) { + if (listed.insert(id).second) { + channels.push_back(&tracks.by_id.at(id)); + } + } + // A sidecar from before the order was kept lists only the channels keeping keys, and leaves out a + // clip's channel whose keys all sit on its actions' ends. + for (const auto &[id, track] : tracks.by_id) { + if (!listed.contains(id) && !track.keys.empty()) { + channels.push_back(&track); + } + } + Record record; + std::size_t at = 0; + for (std::size_t c = 0; c < channels.size(); c++) { + const Track &track = *channels[c]; + record.Word(at, track.frame); + record.Word(at + 4, track.target); + record.Word(at + 8, static_cast(track.type)); + record.Word(at + 12, track.reserved[0]); + record.Word(at + 16, static_cast(track.keys.size())); + record.Word(at + 20, c + 1 < channels.size() ? static_cast(32 + track.keys.size() * 32) : 0); + record.Word(at + 24, track.reserved[1]); + record.Word(at + 28, track.reserved[2]); + at += 32; + for (const auto &[frame, key] : track.keys) { + record.Word(at, frame); + for (std::size_t w = 0; w < 3; w++) { + record.Word(at + 4 + w * 4, key.reserved[w]); + } + for (std::size_t w = 0; w < 4; w++) { + record.Word(at + 16 + w * 4, key.value[w]); + } + at += 32; + } + } + return record.Take(); +} + +// Rebuilds a motion a scene's own configuration links to it, published beside the scene: its +// clip, named after the motion, covers it from frame 0, and the sidecar keeps the rest. +std::vector SceneMotion(const Bundle &bundle, std::string_view alias) { + Tracks tracks; + Whole whole; + Json own = Json::array(); + for (const Json &channel : bundle.sidecar.at("motion_channels")) { + if (Key(channel.value("alias", "")) == Key(alias)) { + own.push_back(channel); + } + } + AddResidue(tracks, whole, own); + std::string_view leaf = alias.substr(alias.rfind('/') + 1); + const std::string clip_name = std::string(leaf.substr(0, leaf.rfind('.'))) + "#0"; + for (std::size_t a = 0; a < bundle.asset.animations.size(); a++) { + if (std::string_view(bundle.asset.animations[a].name) == clip_name) { + const fastgltf::Animation &animation = bundle.asset.animations[a]; + std::uint32_t end = 0; + for (const fastgltf::AnimationSampler &sampler : animation.samplers) { + fastgltf::iterateAccessor(bundle.asset, bundle.asset.accessors[sampler.inputAccessor], [&](float time) { + end = std::max(end, static_cast(std::lround(time * 30.0f))); + }); + } + AddClip(tracks, bundle, bundle.sidecar, 0, a, 0, end, whole); + break; + } + } + return Write(tracks); +} + +const Json &ModelOf(const Bundle &bundle, const Json &entry) { + if (bundle.sidecar.value("kind", "") == "character") { + return bundle.sidecar.at("models").at(entry.at("index").get()); + } + return bundle.sidecar; +} + +std::unique_ptr> Build(State &state, std::string_view alias, const Json &entry) { + const std::shared_ptr holder = LoadBundle(state, entry.at("bundle").get()); + const Bundle &bundle = *holder; + const std::string &kind = entry.at("kind").get_ref(); + const bool character = bundle.sidecar.value("kind", "") == "character"; + auto bytes = [](std::vector value) { + return std::make_unique>(std::move(value)); + }; + if (kind == "model") { + return bytes(Scene(bundle, ModelOf(bundle, entry))); + } + // A motion is as long as the disc's, its tail zeros. + auto padded = [&](std::vector motion) { + motion.resize(std::max(motion.size(), entry.value("size", std::size_t{0}))); + return motion; + }; + if (character && kind == "animations") { + return bytes(padded(Motion(state, bundle, bundle.sidecar.at("resources").at(kind).at(entry.at("index").get())))); + } + if (character && (kind == "weights" || kind == "bind_poses")) { + const Json &resource = bundle.sidecar.at("resources").at(kind).at(entry.at("index").get()); + return bytes(kind == "weights" ? Channels(Pointers(resource.at("channels"))) : Matrices(resource.at("matrices"))); + } + if (kind == "weights") { + return bytes(Channels(Pointers(FromSource(bundle.sidecar.at("weights"), alias).at("channels")))); + } + if (kind == "bind_poses") { + return bytes(Matrices(FromSource(bundle.sidecar.at("bind_poses"), alias).at("matrices"))); + } + if (kind == "motion_channels") { + return bytes(padded(SceneMotion(bundle, alias))); + } + return nullptr; +} + +} // namespace + +bool Publishes(std::string_view alias) { + State &state = Shared(); + std::lock_guard lock(state.mutex); + Refresh(state); + return state.index.contains(Key(alias)); +} + +namespace { + +// Rebuild, with the state locked. The bytes live until the next rebuild. +const std::vector *RebuildLocked(State &state, std::string_view alias) { + Refresh(state); + const std::string key = Key(alias); + if (auto found = state.rebuilt.find(key); found != state.rebuilt.end()) { + return found->second.get(); + } + auto entry = state.index.find(key); + // A source that failed once fails again; it isn't parsed anew at every read of its file. + if (entry == state.index.end() || state.failed.contains(key)) { + return nullptr; + } + std::unique_ptr> bytes; + try { + bytes = Build(state, alias, entry->second); + // Models, weights and bind poses come back as the disc has them, or not at all. + const std::string kind = entry->second.value("kind", ""); + if (bytes != nullptr && kind != "animations" && kind != "motion_channels" && entry->second.contains("crc32") && + Crc32(*bytes) != entry->second["crc32"].get()) { + throw std::runtime_error("rebuilt bytes don't match the disc's"); + } + } catch (const std::exception &error) { + std::fprintf(stderr, "%.*s: cannot rebuild from its glTF: %s\n", static_cast(alias.size()), alias.data(), + error.what()); + bytes.reset(); + } + if (bytes == nullptr) { + state.failed.insert(key); + return nullptr; + } + while (!state.rebuilt_order.empty() && state.rebuilt_bytes + bytes->size() > kKeptRebuiltBytes) { + auto oldest = state.rebuilt.find(state.rebuilt_order.front()); + state.rebuilt_bytes -= oldest->second->size(); + state.rebuilt.erase(oldest); + state.rebuilt_order.pop_front(); + } + state.rebuilt_bytes += bytes->size(); + state.rebuilt_order.push_back(key); + return state.rebuilt.emplace(key, std::move(bytes)).first->second.get(); +} + +} // namespace + +const std::vector *Rebuild(std::string_view alias) { + State &state = Shared(); + std::lock_guard lock(state.mutex); + return RebuildLocked(state, alias); +} + +namespace { + +// The place of one pack member in a buffer: its entry, its bytes and the room before the next entry. +struct Place { + std::size_t entry; + std::size_t start; + std::size_t room; +}; + +std::uint32_t Read32(std::span bytes, std::size_t at) { + std::uint32_t value = 0; + std::memcpy(&value, bytes.data() + at, 4); + return value; +} + +// Finds a member in the pack spanning `bytes`, by name or, after "#entry:", by position. +std::optional Member(std::span bytes, std::string_view name) { + std::optional wanted; + if (const std::size_t marker = name.find("#entry:"); marker != std::string_view::npos) { + wanted = std::stoul(std::string(name.substr(marker + 7))); + name = name.substr(0, marker); + } + std::size_t at = 0; + for (std::size_t index = 0; at + 76 <= bytes.size() && bytes[at] != 0; index++) { + const std::size_t offset = Read32(bytes, at + 64); + const std::size_t size = Read32(bytes, at + 68); + const std::size_t next = Read32(bytes, at + 72); + if (offset < 76 || next < 76 || next > bytes.size() - at || offset > next) { + return std::nullopt; + } + const char *entry_name = reinterpret_cast(bytes.data() + at); + const bool match = wanted ? index == *wanted + : Key(std::string_view(entry_name, strnlen(entry_name, 64))) == Key(name); + if (match) { + // A converted member may have been emptied from the data; its room remains. + return Place{at, at + offset, std::max(next - offset, size)}; + } + at += next; + } + return std::nullopt; +} + +} // namespace + +namespace { + +// Writes one source back where the disc has it in `bytes`: each "/member" of `rest` enters a pack, +// "#offset:n" a scene inside the file or member. +void SpliceOne(State &state, const std::string &alias, const Json &entry, const std::string &rest, + std::span bytes) { + std::span place = bytes; + std::optional member; + for (std::size_t at = 0; at < rest.size();) { + const std::size_t end = rest.find('/', at + 1); + const std::string part = rest.substr(at, end == std::string::npos ? std::string::npos : end - at); + at = end == std::string::npos ? rest.size() : end; + std::string name = part.front() == '/' ? part.substr(1) : part; + std::size_t offset = 0; + if (const std::size_t marker = name.find("#offset:"); marker != std::string::npos) { + offset = std::stoul(name.substr(marker + 8)); + name = name.substr(0, marker); + } + if (!name.empty()) { + member = Member(place, name); + if (!member) { + return; + } + place = place.subspan(member->start, std::min(member->room, place.size() - member->start)); + } + if (offset > 0) { + if (offset > place.size()) { + return; + } + place = place.subspan(offset); + member.reset(); + } + } + // Only the bytes the source was converted from, or the zeros stripping left of them, are + // replaced: a file a mod or another release changed keeps what it has. + const std::size_t size = std::min(entry.value("size", place.size()), place.size()); + const auto current = place.first(size); + const bool emptied = std::all_of(current.begin(), current.end(), [](std::uint8_t b) { return b == 0; }); + if (!emptied && entry.contains("crc32") && Crc32(current) != entry["crc32"].get()) { + std::fprintf(stderr, "%s: the data's bytes aren't those its glTF was converted from; kept as read\n", alias.c_str()); + return; + } + const std::vector *rebuilt = RebuildLocked(state, alias); + if (rebuilt == nullptr || rebuilt->size() > place.size()) { + if (emptied && state.stripped) { + Unrecoverable(alias + " can't be rebuilt from its glTF"); + } + if (rebuilt != nullptr) { + std::fprintf(stderr, "%s: rebuilt source is larger than its place, kept as read\n", alias.c_str()); + } + return; + } + // A motion goes in only as the disc has it. The sidecar doesn't yet keep the order of a motion's + // channels, and the game reads every vertex channel at the first channel's key: rebuilt in another + // order, Saget's vertex animation in the attract movie wrote past his mesh and crashed its draw. + const std::string &kind = entry.at("kind").get_ref(); + if ((kind == "animations" || kind == "motion_channels") && entry.contains("crc32") && + Crc32(*rebuilt) != entry["crc32"].get()) { + if (emptied && state.stripped) { + Unrecoverable(alias + " can't be rebuilt from its glTF as the disc has it"); + } + return; + } + std::memcpy(place.data(), rebuilt->data(), rebuilt->size()); + // A pack member's size becomes the rebuilt one; the game reads motions by their own chain. + if (member) { + const std::uint32_t size = static_cast(rebuilt->size()); + std::memcpy(place.data() - (member->start - member->entry) + 68, &size, 4); + } +} + +} // namespace + +void Splice(std::string_view path, std::span bytes) { + State &state = Shared(); + std::lock_guard lock(state.mutex); + Refresh(state); + std::vector> sources; + const std::string file = Key(path); + for (auto it = state.index.lower_bound(file); it != state.index.end() && it->first.starts_with(file); ++it) { + const std::string_view rest = std::string_view(it->first).substr(file.size()); + const std::string kind = it->second.value("kind", ""); + const bool rebuildable = kind == "model" || kind == "animations" || kind == "weights" || + kind == "bind_poses" || kind == "motion_channels"; + if (rebuildable && (rest.empty() || rest.front() == '/' || rest.front() == '#')) { + sources.emplace_back(it->first, std::string(rest), &it->second); + } + } + for (const auto &[alias, rest, entry] : sources) { + // Nothing a source can't be rebuilt for reaches the game: the file keeps what it read. + try { + SpliceOne(state, alias, *entry, rest, bytes); + } catch (const std::exception &error) { + std::fprintf(stderr, "%s: cannot rebuild from its glTF: %s\n", alias.c_str(), error.what()); + } + } +} + +void Reset() { + State &state = Shared(); + std::lock_guard lock(state.mutex); + Forget(state); +} + +} // namespace dc::data::model_gltf diff --git a/port/src/data/model_gltf.hpp b/port/src/data/model_gltf.hpp new file mode 100644 index 000000000..4fdf7a8bd --- /dev/null +++ b/port/src/data/model_gltf.hpp @@ -0,0 +1,39 @@ +#pragma once + +#include +#include +#include +#include + +/** + * Dark Cloud's models, motions, skin weights and bind poses are published as glTF bundles with + * sidecars (docs/MODEL_FRAMEWORK.md). The game's loaders still take the retail records, so the + * records are rebuilt here from a bundle, byte for byte, in place of the disc's. + */ +namespace dc::data::model_gltf { + +/** + * Whether the published data holds the source file `alias`: a logical path, then the pack members + * inside it ("chara/c01d.chr/c01d.mds"), or a scene inside a file ("gedit/s26/scene.scn#offset:4016"). + */ +bool Publishes(std::string_view alias); + +/** + * The source file `alias` rebuilt from its bundle, or null when the data doesn't publish it or the + * bundle can't rebuild it. The bytes live until the next call. + */ +const std::vector *Rebuild(std::string_view alias); + +/** + * Writes every published source inside the file `path`, just read into `bytes`, back where the + * disc has it: pack members (nested packs included) and scenes at offsets, rebuilt from their + * bundles. A source that doesn't fit its place is left as the file has it, and reported. + */ +void Splice(std::string_view path, std::span bytes); + +/** + * Forgets the index, the bundles and every rebuilt file, for when the data changes. + */ +void Reset(); + +} // namespace dc::data::model_gltf diff --git a/port/src/game/cloth.cpp b/port/src/game/cloth.cpp index 6b50827c3..06cbb9767 100644 --- a/port/src/game/cloth.cpp +++ b/port/src/game/cloth.cpp @@ -3,23 +3,35 @@ #include #include +#include +#include #include +#include #include +#include #include +#include +#include +#include +#include #include "bound.hpp" #include "chararead.hpp" #include "frame.hpp" #include "framevu1.hpp" +#include "mdt.hpp" #include "mglib.hpp" +#include "platform/config.hpp" #include "render/draw3d.hpp" #include "renderinfo.hpp" #include "texture.hpp" +#include "visualvu1.hpp" #include "vu0_ops.hpp" #include "wind.hpp" namespace dc::game { +using namespace dc::platform; using namespace dc::render; void CCloth::Draw() { @@ -128,14 +140,504 @@ void Follow(float *vertex, float *velocity, float *world_home, const float *home } } +bool StartsWith(const char *name, const char *prefix) { + return std::strncmp(name, prefix, std::strlen(prefix)) == 0; +} + +using cloth_body::Capsule; + +std::array WorldPosition(CFrame *frame) { + sceVu0FMATRIX matrix; + frame->GetLWMatrix(matrix); + return {matrix[3][0], matrix[3][1], matrix[3][2]}; +} + +// The body as capsules. A limb is a chain with two joints ("jntN_1" to "jntN_2" to "effN", an arm or a +// leg): a capsule along each joint, a fifth of its length thick. The torso is a chain with one joint whose +// end the other chains branch from (the pelvis, the spine up to the shoulders): a capsule along it, 0.4 of +// its length thick until FitTorsos shapes it to the mesh (Toan's chest lies 0.65 to 1.27 from his 3.1-long +// spine). The other chains with one joint (the head, the hands, the feet) go in `others`, when asked for. +void BodyCapsules(CFrame *frame, std::vector &capsules, std::vector *others = nullptr) { + CFrame *root = frame; + while (root->parent != nullptr && root->reference == 0) { + root = root->parent; + } + auto joint_child = [](CFrame *joint) -> CFrame * { + for (CFrame *child = joint->child; child != nullptr; child = child->brother) { + if (StartsWith(child->name, "jnt") || StartsWith(child->name, "eff")) { + return child; + } + } + return nullptr; + }; + std::vector pending = {root}; + while (!pending.empty()) { + CFrame *frame_at = pending.back(); + pending.pop_back(); + for (CFrame *child = frame_at->child; child != nullptr; child = child->brother) { + pending.push_back(child); + } + CFrame *middle = StartsWith(frame_at->name, "jnt") ? joint_child(frame_at) : nullptr; + if (middle == nullptr || StartsWith(frame_at->parent->name, "jnt")) { + continue; + } + if (StartsWith(middle->name, "eff")) { + int branches = 0; + for (CFrame *child = middle->child; child != nullptr; child = child->brother) { + branches += StartsWith(child->name, "chn") ? 1 : 0; + } + if (branches >= 2) { + const std::array a = WorldPosition(frame_at); + const std::array b = WorldPosition(middle); + const float length = std::hypot(b[0] - a[0], b[1] - a[1], b[2] - a[2]); + Capsule torso = {a, b, length * 0.4f, true}; + // The joint's axis least along the spine, square to it: the fitted shape turns with the body. + sceVu0FMATRIX matrix; + frame_at->GetLWMatrix(matrix); + const float axis[3] = {(b[0] - a[0]) / length, (b[1] - a[1]) / length, (b[2] - a[2]) / length}; + int across = 0; + float along = 2.0f; + for (int row = 0; row < 3; row++) { + const float dot = std::fabs(matrix[row][0] * axis[0] + matrix[row][1] * axis[1] + matrix[row][2] * axis[2]) / + std::hypot(matrix[row][0], matrix[row][1], matrix[row][2]); + if (dot < along) { + along = dot; + across = row; + } + } + const float dot = matrix[across][0] * axis[0] + matrix[across][1] * axis[1] + matrix[across][2] * axis[2]; + float side[3]; + for (int k = 0; k < 3; k++) { + side[k] = matrix[across][k] - dot * axis[k]; + } + const float side_length = std::hypot(side[0], side[1], side[2]); + torso.side = {side[0] / side_length, side[1] / side_length, side[2] / side_length}; + capsules.push_back(torso); + } else if (others != nullptr) { + const std::array a = WorldPosition(frame_at); + const std::array b = WorldPosition(middle); + const float length = std::hypot(b[0] - a[0], b[1] - a[1], b[2] - a[2]); + others->push_back({a, b, std::clamp(length * 0.2f, 0.4f, 1.4f)}); + } + continue; + } + // A limb starts at a joint whose next is a joint too. + for (CFrame *joint = frame_at, *next = middle; next != nullptr; joint = next, next = joint_child(next)) { + const std::array a = WorldPosition(joint); + const std::array b = WorldPosition(next); + const float length = std::hypot(b[0] - a[0], b[1] - a[1], b[2] - a[2]); + capsules.push_back({a, b, std::clamp(length * 0.2f, 0.4f, 1.4f)}); + if (!StartsWith(next->name, "jnt")) { + break; + } + } + } +} + +std::array NearestOnCapsule(const Capsule &capsule, const float *p) { + const float ab[3] = {capsule.b[0] - capsule.a[0], capsule.b[1] - capsule.a[1], capsule.b[2] - capsule.a[2]}; + const float length2 = ab[0] * ab[0] + ab[1] * ab[1] + ab[2] * ab[2]; + float t = 0.0f; + if (length2 > 1e-12f) { + t = std::clamp(((p[0] - capsule.a[0]) * ab[0] + (p[1] - capsule.a[1]) * ab[1] + (p[2] - capsule.a[2]) * ab[2]) / length2, + 0.0f, 1.0f); + } + return {capsule.a[0] + ab[0] * t, capsule.a[1] + ab[1] * t, capsule.a[2] + ab[2] * t}; +} + +// The torso capsule whose surface lies nearest `p`, the point of its line nearest `p`, the distance to it +// and how far the surface lies from the line that way, or null when the skeleton has none. +struct TorsoNearest { + const Capsule *capsule = nullptr; + std::array at; + float distance = 1e9f; + float reach = 0.0f; +}; + +TorsoNearest NearestTorso(std::span capsules, const float *p) { + TorsoNearest nearest; + for (const Capsule &capsule : capsules) { + if (capsule.torso) { + const std::array at = NearestOnCapsule(capsule, p); + const float distance = std::hypot(p[0] - at[0], p[1] - at[1], p[2] - at[2]); + const float reach = cloth_body::Reach(capsule, p); + if (nearest.capsule == nullptr || distance - reach < nearest.distance - nearest.reach) { + nearest = {&capsule, at, distance, reach}; + } + } + } + return nearest; +} + +// How far along `ray` (a unit direction) from `origin` it crosses `face`, or -1 where it misses. +float RayHit(const float *origin, const float *ray, const std::array, 3> &face) { + float edge1[3]; + float edge2[3]; + float to[3]; + for (int k = 0; k < 3; k++) { + edge1[k] = face[1][k] - face[0][k]; + edge2[k] = face[2][k] - face[0][k]; + to[k] = origin[k] - face[0][k]; + } + const float cross[3] = {ray[1] * edge2[2] - ray[2] * edge2[1], ray[2] * edge2[0] - ray[0] * edge2[2], + ray[0] * edge2[1] - ray[1] * edge2[0]}; + const float det = edge1[0] * cross[0] + edge1[1] * cross[1] + edge1[2] * cross[2]; + if (std::fabs(det) < 1e-9f) { + return -1.0f; + } + const float u = (to[0] * cross[0] + to[1] * cross[1] + to[2] * cross[2]) / det; + if (u < 0.0f || u > 1.0f) { + return -1.0f; + } + const float q[3] = {to[1] * edge1[2] - to[2] * edge1[1], to[2] * edge1[0] - to[0] * edge1[2], to[0] * edge1[1] - to[1] * edge1[0]}; + const float v = (ray[0] * q[0] + ray[1] * q[1] + ray[2] * q[2]) / det; + if (v < 0.0f || u + v > 1.0f) { + return -1.0f; + } + const float t = (edge2[0] * q[0] + edge2[1] * q[1] + edge2[2] * q[2]) / det; + return t >= 0.0f ? t : -1.0f; +} + +// Where `p` lies round the capsule's axis, from 0 to 1 starting at its side, and along it from a to b. +std::array AroundAndAlong(const Capsule &capsule, const float *p) { + const float ab[3] = {capsule.b[0] - capsule.a[0], capsule.b[1] - capsule.a[1], capsule.b[2] - capsule.a[2]}; + const float length2 = ab[0] * ab[0] + ab[1] * ab[1] + ab[2] * ab[2]; + const float ap[3] = {p[0] - capsule.a[0], p[1] - capsule.a[1], p[2] - capsule.a[2]}; + const float t = length2 > 1e-12f ? (ap[0] * ab[0] + ap[1] * ab[1] + ap[2] * ab[2]) / length2 : 0.0f; + float out[3]; + for (int k = 0; k < 3; k++) { + out[k] = ap[k] - ab[k] * t; + } + const float length = std::sqrt(length2); + const float axis[3] = {ab[0] / length, ab[1] / length, ab[2] / length}; + const float other[3] = {axis[1] * capsule.side[2] - axis[2] * capsule.side[1], + axis[2] * capsule.side[0] - axis[0] * capsule.side[2], + axis[0] * capsule.side[1] - axis[1] * capsule.side[0]}; + const float x = out[0] * capsule.side[0] + out[1] * capsule.side[1] + out[2] * capsule.side[2]; + const float y = out[0] * other[0] + out[1] * other[1] + out[2] * other[2]; + float around = std::atan2(y, x) / (2.0f * std::numbers::pi_v); + if (around < 0.0f) { + around += 1.0f; + } + return {around, t}; +} + +// The body's mesh in world space, as drawn: the skinned meshes, which keep their vertices for the motions +// to move. A cloth's frame isn't drawn (clothread.cpp's CommandFRAME), its mesh only the cloth's start. +void BodyMesh(CFrame *frame, cloth_body::Mesh &body) { + CFrame *root = frame; + while (root->parent != nullptr && root->reference == 0) { + root = root->parent; + } + std::vector pending = {root}; + while (!pending.empty()) { + CFrame *frame_at = pending.back(); + pending.pop_back(); + const bool drawn = (frame_at->attr.draw_on & 1) != 0; + if (drawn || (frame_at->attr.draw_on & 2) == 0) { + for (CFrame *child = frame_at->child; child != nullptr; child = child->brother) { + pending.push_back(child); + } + } + // A character's frames are CFrameVu1 (dataset.cpp builds them as an array of them). + CVisualVu1 *visual = static_cast(frame_at)->GetVisual(); + if (!drawn || visual == nullptr) { + continue; + } + const auto *model = reinterpret_cast(visual->GetMDTDataAddress()); + if (model == nullptr) { + continue; + } + const auto *local = reinterpret_cast(reinterpret_cast(model) + model->vertex_ofs); + sceVu0FMATRIX matrix; + frame_at->GetLWMatrix(matrix); + const auto first = static_cast(body.vertices.size()); + for (int k = 0; k < model->vertex_num; k++) { + const float point[4] = {local[k][0], local[k][1], local[k][2], 1.0f}; + float world[4]; + vu0::Apply(world, matrix, point); + body.vertices.push_back({world[0], world[1], world[2]}); + } + // The strips as visualvu1.cpp reads them: each a primitive, a count and a material, then for each + // corner its vertex, normal and texture coordinate indices, and colour's when the model has them. + const auto *mesh = reinterpret_cast(reinterpret_cast(model) + model->mesh_ofs); + const int stride = model->colour_ofs > 0 ? 4 : 3; + const u_int *index = mesh + 4; + for (u_int strip = 0; strip < mesh[2]; strip++) { + const u_int prim = index[0]; + const u_int count = index[1]; + index += 3; + auto corner = [&](u_int k) { return index[k * stride]; }; + for (u_int k = 0; k + 2 < count; k += prim == 4 ? 1 : 3) { + const u_int a = corner(k); + const u_int b = corner(k + 1); + const u_int c = corner(k + 2); + if (std::max({a, b, c}) < static_cast(model->vertex_num)) { + body.triangles.push_back({first + a, first + b, first + c}); + } + } + index += count * stride; + } + } +} + +constexpr float kCapsuleMargin = 0.05f; + +// What a cloth takes from the body when it settles on it: each vertex's clearance from each limb, and the +// torsos' shapes. +struct Settled { + std::vector clearances; + std::vector torsos; +}; + +std::unordered_map &SettledOn() { + static std::unordered_map settled; + return settled; +} + +// The Options screen's Enhanced cloth: the PS2's step, then the free vertices pushed out of the body's +// capsules as the step pushes them out of the boxes, so the body no longer passes through the cloth. +// How near each limb a vertex may come, and the torso's shape, are taken when the cloth first steps on the +// body, or after a reset: the cloth is authored resting on the body as it stands. +template +void PushClothOutOfBody(CCloth &cloth, At at) { + std::vector capsules; + std::vector others; + BodyCapsules(cloth.frame, capsules, &others); + const std::size_t count = capsules.size(); + Settled &settled = SettledOn()[&cloth]; + std::vector &clearances = settled.clearances; + if (clearances.size() != static_cast(cloth.num_i * cloth.num_j) * count) { + settled.torsos.clear(); + if (std::ranges::any_of(capsules, &Capsule::torso)) { + cloth_body::Mesh body; + BodyMesh(cloth.frame, body); + cloth_body::FitTorsos(capsules, others, body); + std::ranges::copy_if(capsules, std::back_inserter(settled.torsos), &Capsule::torso); + } + clearances.resize(static_cast(cloth.num_i * cloth.num_j) * count); + for (int i = 0; i < cloth.num_i; i++) { + for (int j = 0; j < cloth.num_j; j++) { + for (std::size_t c = 0; c < count; c++) { + clearances[(i * cloth.num_j + j) * count + c] = cloth_body::Clearance(capsules[c], cloth.world_home[i][j]); + } + } + } + } + auto torso = settled.torsos.begin(); + for (Capsule &capsule : capsules) { + if (capsule.torso && torso != settled.torsos.end()) { + capsule.shape = torso->shape; + capsule.fitted = torso->fitted; + ++torso; + } + } + for (int i = 0; i < cloth.num_i; i++) { + for (int j = 1; j < cloth.num_j; j++) { + cloth_body::PushOutOfBody(capsules, std::span(clearances).subspan((i * cloth.num_j + j) * count, count), + cloth.world_home[i][j], at(i, j)); + } + } +} + } // namespace +float cloth_body::Reach(const Capsule &capsule, const float *p) { + if (!capsule.fitted) { + return capsule.radius; + } + const auto [around, along] = AroundAndAlong(capsule, p); + const float sector = around * kTorsoSectors - 0.5f; + const float band = std::clamp(along * kTorsoBands - 0.5f, 0.0f, static_cast(kTorsoBands - 1)); + const int sector0 = static_cast(std::floor(sector)); + const int band0 = std::min(static_cast(band), kTorsoBands - 2); + const float s = sector - static_cast(sector0); + const float h = std::min(band - static_cast(band0), 1.0f); + auto at = [&](int sector_at, int band_at) { + return capsule.shape[std::min(band_at, kTorsoBands - 1) * kTorsoSectors + (sector_at + kTorsoSectors) % kTorsoSectors]; + }; + const float low = at(sector0, band0) * (1.0f - s) + at(sector0 + 1, band0) * s; + const float high = at(sector0, band0 + 1) * (1.0f - s) + at(sector0 + 1, band0 + 1) * s; + return low * (1.0f - h) + high * h; +} + +void cloth_body::FitTorsos(std::span capsules, std::span others, const Mesh &body) { + // Too few vertices, or faces in too few directions, to say what shape a torso is, and it keeps its + // round one. + constexpr int kFewest = 8; + constexpr int kFewestCrossed = kTorsoSectors * kTorsoBands / 2; + + std::vector owners(body.vertices.size(), capsules.size()); + std::vector counts(capsules.size()); + for (std::size_t v = 0; v < body.vertices.size(); v++) { + const float *vertex = body.vertices[v].data(); + auto outside = [&](const Capsule &capsule) { + const std::array at = NearestOnCapsule(capsule, vertex); + return std::hypot(vertex[0] - at[0], vertex[1] - at[1], vertex[2] - at[2]) - capsule.radius; + }; + std::size_t owner = capsules.size(); + float nearest = 1e9f; + for (std::size_t c = 0; c < capsules.size(); c++) { + if (outside(capsules[c]) < nearest) { + nearest = outside(capsules[c]); + owner = c; + } + } + if (owner < capsules.size() && capsules[owner].torso && + std::ranges::none_of(others, [&](const Capsule &other) { return outside(other) < nearest; })) { + owners[v] = owner; + counts[owner]++; + } + } + for (std::size_t c = 0; c < capsules.size(); c++) { + Capsule &torso = capsules[c]; + if (!torso.torso || counts[c] < kFewest) { + continue; + } + // The torso's faces: those with most of their corners on it, not an arm's or a hand's. + std::vector, 3>> faces; + for (const std::array &triangle : body.triangles) { + const int on = static_cast(std::ranges::count_if(triangle, [&](std::uint32_t v) { return owners[v] == c; })); + if (on >= 2) { + faces.push_back({body.vertices[triangle[0]], body.vertices[triangle[1]], body.vertices[triangle[2]]}); + } + } + const float length = std::hypot(torso.b[0] - torso.a[0], torso.b[1] - torso.a[1], torso.b[2] - torso.a[2]); + const float axis[3] = {(torso.b[0] - torso.a[0]) / length, (torso.b[1] - torso.a[1]) / length, + (torso.b[2] - torso.a[2]) / length}; + const float other[3] = {axis[1] * torso.side[2] - axis[2] * torso.side[1], axis[2] * torso.side[0] - axis[0] * torso.side[2], + axis[0] * torso.side[1] - axis[1] * torso.side[0]}; + // Each direction and height looks out from the axis to the furthest face it crosses. + std::array shape; + shape.fill(-1.0f); + for (int band = 0; band < kTorsoBands; band++) { + const float along = (static_cast(band) + 0.5f) / static_cast(kTorsoBands); + float origin[3]; + for (int k = 0; k < 3; k++) { + origin[k] = torso.a[k] + (torso.b[k] - torso.a[k]) * along; + } + for (int sector = 0; sector < kTorsoSectors; sector++) { + const float angle = 2.0f * std::numbers::pi_v * (static_cast(sector) + 0.5f) / kTorsoSectors; + float ray[3]; + for (int k = 0; k < 3; k++) { + ray[k] = torso.side[k] * std::cos(angle) + other[k] * std::sin(angle); + } + float &cell = shape[band * kTorsoSectors + sector]; + for (const auto &face : faces) { + cell = std::max(cell, RayHit(origin, ray, face)); + } + } + } + if (std::ranges::count_if(shape, [](float reach) { return reach >= 0.0f; }) < kFewestCrossed) { + continue; + } + // A direction no face lies in takes the shape between its neighbours round the band; a band with + // none takes the nearest band's. + for (int band = 0; band < kTorsoBands; band++) { + float *row = &shape[band * kTorsoSectors]; + std::array filled; + std::copy(row, row + kTorsoSectors, filled.begin()); + for (int sector = 0; sector < kTorsoSectors; sector++) { + if (row[sector] >= 0.0f) { + continue; + } + int before = 1; + while (before < kTorsoSectors && row[(sector - before + kTorsoSectors) % kTorsoSectors] < 0.0f) { + before++; + } + if (before == kTorsoSectors) { + break; + } + int after = 1; + while (row[(sector + after) % kTorsoSectors] < 0.0f) { + after++; + } + const float from = row[(sector - before + kTorsoSectors) % kTorsoSectors]; + const float to = row[(sector + after) % kTorsoSectors]; + filled[sector] = from + (to - from) * static_cast(before) / static_cast(before + after); + } + std::copy(filled.begin(), filled.end(), row); + } + for (int band = 0; band < kTorsoBands; band++) { + for (int step = 1; shape[band * kTorsoSectors] < 0.0f && step < kTorsoBands; step++) { + for (const int from : {band - step, band + step}) { + if (from >= 0 && from < kTorsoBands && shape[from * kTorsoSectors] >= 0.0f) { + std::copy_n(&shape[from * kTorsoSectors], kTorsoSectors, &shape[band * kTorsoSectors]); + break; + } + } + } + } + torso.shape = shape; + torso.fitted = true; + } +} + +float cloth_body::Clearance(const Capsule &limb, const float *home) { + const std::array at = NearestOnCapsule(limb, home); + return std::min(limb.radius + kCapsuleMargin, std::hypot(home[0] - at[0], home[1] - at[1], home[2] - at[2])); +} + +// The PS2's step lets the cloth through the chest when the body turns quickly (an about-turn swings +// Toan's chest through the poncho over a dozen ticks) and when an arm presses it there. Carrying the cloth +// round with the turn instead took most of its swing away; keeping it off the chest keeps the swing. +// Squeezed between an arm and the chest, the cloth stays on the chest and the arm overlaps it. +void cloth_body::PushOutOfBody(std::span capsules, std::span clearances, const float *home, + float *vertex) { + for (const Capsule &capsule : capsules) { + if (!capsule.torso) { + continue; + } + // Cloth authored resting inside the torso (Toan's collar) goes no deeper than it rests: pushed out + // to the surface, it puffed; left be, it sank deeper into his chest in the attract movie's poses. + const std::array home_at = NearestOnCapsule(capsule, home); + const float rest = std::hypot(home[0] - home_at[0], home[1] - home_at[1], home[2] - home_at[2]); + const float reach = std::min(cloth_body::Reach(capsule, vertex) + kCapsuleMargin, rest); + const std::array nearest = NearestOnCapsule(capsule, vertex); + const float out[3] = {vertex[0] - nearest[0], vertex[1] - nearest[1], vertex[2] - nearest[2]}; + const float distance = std::hypot(out[0], out[1], out[2]); + if (distance < reach && distance > 1e-6f) { + for (int axis = 0; axis < 3; axis++) { + vertex[axis] = nearest[axis] + out[axis] * reach / distance; + } + } + } + for (std::size_t c = 0; c < capsules.size(); c++) { + const Capsule &capsule = capsules[c]; + if (capsule.torso) { + continue; + } + const float reach = c < clearances.size() ? clearances[c] : capsule.radius + kCapsuleMargin; + const std::array nearest = NearestOnCapsule(capsule, vertex); + const float out[3] = {vertex[0] - nearest[0], vertex[1] - nearest[1], vertex[2] - nearest[2]}; + const float distance = std::hypot(out[0], out[1], out[2]); + if (distance >= reach || distance <= 1e-6f) { + continue; + } + float pushed[3]; + for (int axis = 0; axis < 3; axis++) { + pushed[axis] = nearest[axis] + out[axis] * reach / distance; + } + const TorsoNearest torso = NearestTorso(capsules, pushed); + if (torso.capsule != nullptr && torso.distance > 1e-6f) { + const float floor = std::min(torso.reach + kCapsuleMargin, NearestTorso(capsules, home).distance); + if (torso.distance < floor) { + for (int axis = 0; axis < 3; axis++) { + pushed[axis] = torso.at[axis] + (pushed[axis] - torso.at[axis]) * floor / torso.distance; + } + } + } + std::copy(pushed, pushed + 3, vertex); + } +} + void CCloth::Step(int step) { if (num_i <= 0 || num_i > 16 || num_j <= 0 || num_j > 16) { return; } if (step < 0) { + SettledOn().erase(this); Clear(); return; } @@ -265,6 +767,13 @@ void CCloth::Step(int step) { } } + if (ConfigGet().enhanced_cloth && frame != nullptr && !reset) { + PushClothOutOfBody(*this, work_at); + } else if (reset) { + // A cut or a teleport puts the cloth back on its rest shape, on the body as it stands now. + SettledOn().erase(this); + } + sceVu0FVECTOR wind_force = {0.0f, 0.0f, 0.0f, 0.0f}; for (int i = 0; i < num_i; i++) { diff --git a/port/src/options/rows.cpp b/port/src/options/rows.cpp index 2b447bfd9..61ef167b2 100644 --- a/port/src/options/rows.cpp +++ b/port/src/options/rows.cpp @@ -649,6 +649,9 @@ constexpr Row kDisplayRows[] = { "\"Anisotropic Filter\"\nSharper textures on\nsurfaces seen at a\nslant.", AnisotropyCount, AnisotropyChoice, SetAnisotropy, nullptr, "Off|2x|4x|8x|16x"), GameRow<&ConfigGameOptions::soft_focus, true>("video.soft_focus", "Soft Focus", "On|Off", 0x169), + NamedRow<&Config::enhanced_cloth>("video.cloth", "Cloth", + "\"Cloth\"\nOriginal: the PS2's.\nEnhanced: the body no\nlonger passes through.", + "Original|Enhanced"), }; #ifdef __ANDROID__ @@ -666,6 +669,7 @@ constexpr Row kAndroidDisplayRows[] = { kDisplayRows[9], kDisplayRows[10], kDisplayRows[11], + kDisplayRows[12], }; #elifdef __SWITCH__ constexpr Row kSwitchDisplayRows[] = { @@ -674,6 +678,7 @@ constexpr Row kSwitchDisplayRows[] = { kDisplayRows[8], kDisplayRows[9], kDisplayRows[10], + kDisplayRows[11], }; #endif diff --git a/port/src/platform/config.cpp b/port/src/platform/config.cpp index b0654574c..fbd5a5d2d 100644 --- a/port/src/platform/config.cpp +++ b/port/src/platform/config.cpp @@ -609,6 +609,9 @@ bool Apply(Config &config, std::string_view name, const Json &value) { if (name == "discord.rich_presence") { return ReadBool(value, config.discord_rich_presence); } + if (name == "video.cloth") { + return ReadChoice(value, "original", "enhanced", config.enhanced_cloth); + } if (name == "audio.soundtrack") { return ReadChoice(value, "ps2", "custom", config.soundtrack); } @@ -770,6 +773,7 @@ std::string ConfigSerialize(const Config &config) { root["video"]["shadow_distance"] = Shortest(config.shadow_distance); root["video"]["anisotropy"] = config.anisotropy; root["video"]["soft_focus"] = options.soft_focus; + root["video"]["cloth"] = config.enhanced_cloth ? "enhanced" : "original"; root["audio"]["master_volume"] = Shortest(config.master_volume); root["audio"]["sound"] = SoundName(config); root["audio"]["soundtrack"] = config.soundtrack ? "custom" : "ps2"; diff --git a/port/src/platform/config.hpp b/port/src/platform/config.hpp index 943ffbaf0..b06a678d4 100644 --- a/port/src/platform/config.hpp +++ b/port/src/platform/config.hpp @@ -67,6 +67,7 @@ struct Config : HostSettings { float detail_distance = 0.0f; float shadow_distance = 0.0f; int anisotropy = 0; + bool enhanced_cloth = false; /**< video.cloth: false runs the PS2's cloth, true also keeps it out of the body. */ float mouse_camera_return = 0.2f; /**< Third-person vertical return after mouse input: 0 holds height, 1 is retail's rate. */ ConfigGameOptions options; bool discord_rich_presence = true; diff --git a/port/src/tests/character_profile_test.cpp b/port/src/tests/character_profile_test.cpp new file mode 100644 index 000000000..59958c552 --- /dev/null +++ b/port/src/tests/character_profile_test.cpp @@ -0,0 +1,119 @@ +#include +#include + +#include +#include + +#include "../../../tools/dcdata/character_profile.hpp" + +namespace dc::test { + +namespace { + +using nlohmann::json; + +json Read(std::string_view alias, std::string_view text) { + return json::parse(dcdata::profile::Read(alias, text)); +} + +std::vector Commands(const json &document, std::string_view name) { + std::vector found; + for (const json &command : document["commands"]) { + if (command["command"] == name) { + found.push_back(command); + } + } + return found; +} + +// 立ち, 走り and バトル （元気） in Shift-JIS, as the disc's comments are written. +constexpr std::string_view kStand = "\x97\xa7\x82\xbf"; +constexpr std::string_view kRun = "\x91\x96\x82\xe8"; +constexpr std::string_view kBattle = "\x83\x6f\x83\x67\x83\x8b\x20\x81\x69\x8c\xb3\x8b\x43\x81\x6a"; + +std::string Hex(std::string_view bytes) { + std::string out; + for (unsigned char c : bytes) { + out += std::format("{:02x}", c); + } + return out; +} + +TEST(CharacterProfile, KeysTakeTheirNamesFromTheLineCommentAndKeepTheirFrames) { + const std::string cfg = std::string("MODEL \"c01d.mds\"\r\n") + + "MOTION 0, \"c01dbas.mot\", \"c01dbas.bbp\", \"c01dbas.wgt\"\r\n" + + "KEY_START 0\r\n" + + "//KEY start key, end key, step\r\n" + + "KEY\t10,\t20,\t0.05\t//" + std::string(kStand) + "\t\t0\r\n" + + "KEY\t70,\t90,\t0.6,\t//" + std::string(kRun) + "\t1\r\n" + + "KEY\t229,\t250\t0.3\t//" + std::string(kBattle) + " 2\r\n" + + "KEY\t300,\t310,\t0.5\t//\t14\r\n" + + "KEY\t320,\t330,\t0.5\r\n" + + "MOTION_END\r\n"; + const json document = Read("chara/c01d.chr/info.cfg", cfg); + const std::vector keys = Commands(document, "KEY"); + ASSERT_EQ(keys.size(), 5u); + EXPECT_EQ(keys[0]["args"][0]["bytes"], Hex(kStand)); + EXPECT_EQ(keys[0]["args"][1]["value"], 10); + EXPECT_EQ(keys[0]["args"][2]["value"], 20); + EXPECT_FLOAT_EQ(keys[0]["args"][3]["value"].get(), 0.05f); + EXPECT_EQ(keys[1]["args"][0]["bytes"], Hex(kRun)); + // A missing comma still separates, as the game's reader splits words on blanks too; a name + // keeps the blank inside it. + EXPECT_EQ(keys[2]["args"][0]["bytes"], Hex(kBattle)); + EXPECT_EQ(keys[2]["args"][2]["value"], 250); + // A comment that is only the index, or none, leaves the key unnamed. + EXPECT_EQ(keys[3]["args"][0]["type"], "none"); + EXPECT_EQ(keys[4]["args"][0]["type"], "none"); + const json motion = Commands(document, "MOTION").at(0); + ASSERT_EQ(motion["args"].size(), 4u); + EXPECT_EQ(motion["args"][3]["bytes"], Hex("c01dbas.wgt")); +} + +TEST(CharacterProfile, TextureAnimationsKeepTheirPlaceAmongTheCommands) { + const std::string cfg = "IMG 0,\"c01d01.img\"\nIMG_END\nTEX_ANIME 1,1\n" + "TEX_ANIME_DATA \"c01d11\",0,0,128,48,\"c01d01\",0,27,100,0\t//eyes\n" + "TEX_ANIME_END\nMATERIAL_ANIME 0\nVERTEX_ANIME 1\nMODEL \"c01d.mds\"\n"; + const json document = Read("info.cfg", cfg); + std::vector order; + for (const json &command : document["commands"]) { + order.push_back(command["command"]); + } + EXPECT_EQ(order, (std::vector{"IMG", "IMG_END", "TEX_ANIME", "TEX_ANIME_DATA", "TEX_ANIME_END", + "VERTEX_ANIME", "MODEL"})); + const json data = Commands(document, "TEX_ANIME_DATA").at(0); + ASSERT_EQ(data["args"].size(), 10u); + EXPECT_EQ(data["args"][5]["bytes"], Hex("c01d01")); + EXPECT_EQ(data["args"][8]["value"], 100); +} + +TEST(CharacterProfile, OtherConfigurationsAreNotProfiles) { + EXPECT_EQ(dcdata::profile::Read("dun/d05main_a.cfg", "PT_BASE \"d05g01_0.mds\",0\nPT_COLS \"d05g01_a.mds\",0\n"), + "null"); + EXPECT_EQ(dcdata::profile::Read("chara/c01d.chr/c01d.mds", "MODEL \"x.mds\"\n"), "null"); +} + +TEST(CharacterProfile, ClothReadsItsArgumentsAcrossLines) { + const std::string clo = "SIZE\t4,\t5\nFRAME\t\"cloth1\"\nK\t0.08,\t0.08,\t0.08\nBOUND\t\"jnt12_1\"\n" + "\t0.0,\t1.0,\t0.0\n\t2.15,\t-0.45,\t1.0\n\t2.15,\t-0.45,\t-1.0\n" + "\t3.66,\t3.31,\t2.73\n\t0.7\n"; + const json document = Read("chara/c01d.chr/turban.clo", clo); + const json bound = Commands(document, "BOUND").at(0); + ASSERT_EQ(bound["args"].size(), 14u); + EXPECT_FLOAT_EQ(bound["args"][5]["value"].get(), -0.45f); + EXPECT_FLOAT_EQ(bound["args"][13]["value"].get(), 0.7f); + EXPECT_EQ(Commands(document, "SIZE").at(0)["args"][1]["value"], 5); +} + +TEST(CharacterProfile, ArgumentsFollowTheGamesChecks) { + // ReadInfo takes no sign in an integer, and runs nothing for a command cut off by the end of + // the file. + const json document = Read("info.cfg", "MODEL \"a.mds\"\nKEY_START -1\nFOOT 211,-1\nBODY_SIZE 1,2"); + EXPECT_EQ(Commands(document, "KEY_START").at(0)["args"][0]["type"], "invalid"); + EXPECT_FLOAT_EQ(Commands(document, "FOOT").at(0)["args"][1]["value"].get(), -1.0f); + EXPECT_TRUE(Commands(document, "BODY_SIZE").empty()); +} + +} // namespace + +} // namespace dc::test diff --git a/port/src/tests/cloth_enhanced_test.cpp b/port/src/tests/cloth_enhanced_test.cpp new file mode 100644 index 000000000..7a7711a93 --- /dev/null +++ b/port/src/tests/cloth_enhanced_test.cpp @@ -0,0 +1,354 @@ +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "cloth.hpp" +#include "frame.hpp" +#include "platform/config.hpp" +#include "platform/paths.hpp" +#include "platform_fixture.hpp" + +namespace dc::test { + +using namespace dc::game; +using namespace dc::platform; + +namespace { + +// A body with one leg, a joint and its end, and a cloth hanging in front of it from a frame of its own. +struct Body { + CFrame root; + CFrame chain; + CFrame hip; + CFrame knee; + CFrame end; + CFrame hang; + + Body() { + for (CFrame *frame : {&root, &chain, &hip, &knee, &end, &hang}) { + frame->Initialize(); + } + std::strcpy(root.name, "body"); + std::strcpy(chain.name, "chn5"); + std::strcpy(hip.name, "jnt5_1"); + std::strcpy(knee.name, "jnt5_2"); + std::strcpy(end.name, "eff5"); + std::strcpy(hang.name, "cloth1"); + chain.SetParent(&root); + hip.SetParent(&chain); + knee.SetParent(&hip); + end.SetParent(&knee); + hang.SetParent(&root); + hip.local[3][1] = -2.0f; + knee.local[3][1] = -2.0f; + end.local[3][1] = -2.0f; + } +}; + +// The leg runs from (0, -2, 0) through the knee to (0, -6, 0); each half's capsule is 0.2 of its length +// thick, at least 0.4. +constexpr float kLegRadius = 0.4f; + +float FromLeg(const float *point) { + const float y = std::clamp(point[1], -6.0f, -2.0f); + return std::hypot(point[0], point[1] - y, point[2]); +} + +// A 5 × 6 cloth in front of the leg, pulled back through it by a strong wind-like gravity. +std::unique_ptr HangCloth(CFrame &hang) { + auto cloth = std::make_unique(16, 16, 1.0f); + cloth->num_i = 5; + cloth->num_j = 6; + cloth->frame = &hang; + cloth->bound = nullptr; + cloth->wind = nullptr; + cloth->floor_on = 0; + cloth->stop = 0; + cloth->normal_scale = 1.0f; + for (int axis = 0; axis < 3; axis++) { + cloth->stiffness[axis] = 0.005f; + cloth->follow[axis] = 0.0f; + } + cloth->gravity[0] = 0.0f; + cloth->gravity[1] = -0.05f; + cloth->gravity[2] = -0.2f; + for (int i = 0; i < 5; i++) { + for (int j = 0; j < 6; j++) { + cloth->home[i][j][0] = static_cast(i) - 2.0f; + cloth->home[i][j][1] = -static_cast(j); + cloth->home[i][j][2] = 2.2f; + cloth->home[i][j][3] = 1.0f; + cloth->mask[i][j] = 0; + cloth->rest[i][j][0] = 1.0f; + cloth->rest[i][j][1] = 1.0f; + } + } + cloth->position[0] = 0.0f; + cloth->position[1] = 0.0f; + cloth->position[2] = 2.2f; + cloth->position[3] = 1.0f; + cloth->Step(-1); + return cloth; +} + +float Nearest(const CCloth &cloth, float (*from)(const float *) = FromLeg) { + float nearest = 1e9f; + for (int i = 0; i < cloth.num_i; i++) { + for (int j = 0; j < cloth.num_j; j++) { + nearest = std::min(nearest, from(cloth.point[i][j])); + } + } + return nearest; +} + +struct ConfigScope { + std::filesystem::path root = std::filesystem::temp_directory_path() / + ("chronicle-cloth-test-" + std::to_string(dc::test::ProcessId())); + + explicit ConfigScope(bool enhanced) { + PathsSetSaveRoot(root); + Config config; + config.enhanced_cloth = enhanced; + EXPECT_TRUE(ConfigChange(config)); + } + + ~ConfigScope() { + ConfigChange(Config{}); + std::error_code error; + std::filesystem::remove_all(root, error); + } +}; + +// A torso along y from (0, -1, 0) to (0, -5, 0), 1.6 thick, and an arm across in front of it along x at +// height -3, its axis at the given depth, 0.6 thick. +std::vector ArmBeforeChest(float arm_depth = 2.0f, bool torso = true) { + std::vector capsules = { + {{-3.0f, -3.0f, arm_depth}, {3.0f, -3.0f, arm_depth}, 0.6f} + }; + if (torso) { + capsules.push_back({ + {0.0f, -1.0f, 0.0f}, + {0.0f, -5.0f, 0.0f}, + 1.6f, true + }); + } + return capsules; +} + +// Rests high on the chest, well away from the arm. +constexpr float kHome[3] = {0.0f, -1.5f, 2.5f}; + +// Pushes `vertex` out of `capsules`, with the limbs' clearances taken when the cloth settled on the body +// as `settled` has it (the same body unless a limb has moved since). +void Push(const std::vector &capsules, const std::vector &settled, const float *home, + float *vertex) { + std::vector clearances; + for (const cloth_body::Capsule &capsule : settled) { + clearances.push_back(capsule.torso ? 0.0f : cloth_body::Clearance(capsule, home)); + } + cloth_body::PushOutOfBody(capsules, clearances, home, vertex); +} + +void Push(const std::vector &capsules, const float *home, float *vertex) { + Push(capsules, capsules, home, vertex); +} + +} // namespace + +TEST(ClothEnhanced, KeepsTheClothOutOfTheBody) { + ConfigScope config(true); + Body body; + auto cloth = HangCloth(body.hang); + for (int step = 0; step < 120; step++) { + cloth->Step(0); + EXPECT_GE(Nearest(*cloth), kLegRadius - 1e-3f) << "step " << step; + } + // Pinned along its first column, where the frame holds it. + for (int i = 0; i < cloth->num_i; i++) { + EXPECT_NEAR(cloth->point[i][0][2], 2.2f, 1e-4f); + } +} + +TEST(ClothEnhanced, TheOriginalClothPassesThroughIt) { + ConfigScope config(false); + Body body; + auto cloth = HangCloth(body.hang); + float nearest = 1e9f; + for (int step = 0; step < 120; step++) { + cloth->Step(0); + nearest = std::min(nearest, Nearest(*cloth)); + } + EXPECT_LT(nearest, kLegRadius); +} + +TEST(ClothEnhanced, PushesAVertexOutOfAnArm) { + float vertex[3] = {0.0f, -3.0f, 2.3f}; + Push(ArmBeforeChest(), kHome, vertex); + EXPECT_NEAR(vertex[2], 2.65f, 1e-4f); +} + +TEST(ClothEnhanced, AnArmPressesTheClothOnlyAsFarAsTheChest) { + // Between the arm and the chest: pushed in, onto the chest's surface and no further. + float vertex[3] = {0.0f, -3.0f, 1.8f}; + Push(ArmBeforeChest(), kHome, vertex); + EXPECT_NEAR(vertex[0], 0.0f, 1e-4f); + EXPECT_NEAR(vertex[1], -3.0f, 1e-4f); + EXPECT_NEAR(vertex[2], 1.65f, 1e-4f); +} + +TEST(ClothEnhanced, AnArmNeverPressesTheClothDeeperThanItRests) { + // Resting inside the chest's capsule, and sunk nearer than that: back to its rest depth, and the arm + // in front doesn't press it past it. + const float home[3] = {0.0f, -1.5f, 1.6f}; + float vertex[3] = {0.0f, -3.0f, 1.5f}; + Push(ArmBeforeChest(1.7f), home, vertex); + EXPECT_NEAR(vertex[2], 1.6f, 1e-4f); +} + +TEST(ClothEnhanced, KeepsTheClothOffTheChest) { + float vertex[3] = {0.0f, -3.0f, 0.5f}; + Push(ArmBeforeChest(6.0f), kHome, vertex); + EXPECT_NEAR(vertex[2], 1.65f, 1e-4f); +} + +TEST(ClothEnhanced, ClothRestingInsideTheChestGoesNoDeeperThanItRests) { + // Resting 1.2 from the spine, inside the chest's 1.65: sunk to 0.5, it goes back to 1.2, not 1.65. + const float home[3] = {0.0f, -1.5f, 1.2f}; + float vertex[3] = {0.0f, -3.0f, 0.5f}; + Push(ArmBeforeChest(6.0f), home, vertex); + EXPECT_NEAR(vertex[2], 1.2f, 1e-4f); + float resting[3] = {0.0f, -3.0f, 1.2f}; + Push(ArmBeforeChest(6.0f), home, resting); + EXPECT_NEAR(resting[2], 1.2f, 1e-6f); +} + +TEST(ClothEnhanced, ClothRestingAgainstAnArmGoesNoDeeperThanItRests) { + // Resting 0.3 from the arm's axis, inside its 0.65: pressed to 0.2, it goes back to 0.3, not 0.65. + const float home[3] = {0.0f, -3.0f, 2.3f}; + float vertex[3] = {0.0f, -3.0f, 1.8f}; + Push(ArmBeforeChest(), home, vertex); + EXPECT_NEAR(vertex[2], 1.7f, 1e-4f); + float resting[3] = {home[0], home[1], home[2]}; + Push(ArmBeforeChest(), home, resting); + EXPECT_NEAR(resting[2], 2.3f, 1e-6f); +} + +TEST(ClothEnhanced, AnArmRaisedThroughWhereTheClothRestsPushesIt) { + // The cloth settled with the arm far off; the arm then comes up through where the cloth rests. + const float home[3] = {0.0f, -3.0f, 2.5f}; + float vertex[3] = {0.0f, -3.0f, 2.5f}; + Push(ArmBeforeChest(2.4f), ArmBeforeChest(6.0f), home, vertex); + EXPECT_NEAR(vertex[2], 3.05f, 1e-4f); +} + +TEST(ClothEnhanced, WithoutATorsoAnArmPushesAllTheWay) { + float vertex[3] = {0.0f, -3.0f, 1.8f}; + Push(ArmBeforeChest(2.0f, false), kHome, vertex); + EXPECT_NEAR(vertex[2], 1.35f, 1e-4f); +} + +namespace { + +// A box from `low` to `high`, as twelve triangles. +void AddBox(cloth_body::Mesh &mesh, std::array low, std::array high) { + const auto first = static_cast(mesh.vertices.size()); + for (int corner = 0; corner < 8; corner++) { + mesh.vertices.push_back({(corner & 1) ? high[0] : low[0], (corner & 2) ? high[1] : low[1], (corner & 4) ? high[2] : low[2]}); + } + // Each side, as its two corners varying. + for (const auto &[fixed, u, v] : { + std::array{0, 1, 2}, + std::array{1, 0, 2}, + std::array{2, 0, 1} + }) { + for (int at : {0, 1}) { + auto index = [&](int du, int dv) { + return first + static_cast((at << fixed) | (du << u) | (dv << v)); + }; + mesh.triangles.push_back({index(0, 0), index(1, 0), index(1, 1)}); + mesh.triangles.push_back({index(0, 0), index(1, 1), index(0, 1)}); + } + } +} + +// A torso along y from (0, -1, 0) to (0, -5, 0), 1.6 thick before it is fitted, its side towards +x. +cloth_body::Capsule Torso() { + cloth_body::Capsule torso = { + {0.0f, -1.0f, 0.0f}, + {0.0f, -5.0f, 0.0f}, + 1.6f, true + }; + torso.side = {1.0f, 0.0f, 0.0f}; + return torso; +} + +// A body round that torso 2 wide, its chest 1.2 in front of the spine (+z) and its back 0.6 behind. +cloth_body::Mesh Chest() { + cloth_body::Mesh mesh; + AddBox(mesh, {-1.0f, -5.5f, -0.6f}, {1.0f, -0.5f, 1.2f}); + return mesh; +} + +float ReachTowards(const cloth_body::Capsule &torso, float x, float z) { + const float point[3] = {x, -3.0f, z}; + return cloth_body::Reach(torso, point); +} + +} // namespace + +TEST(ClothEnhanced, FitsTheTorsoToTheBody) { + std::vector capsules = {Torso()}; + cloth_body::FitTorsos(capsules, {}, Chest()); + ASSERT_TRUE(capsules[0].fitted); + // Between its sectors the shape runs straight from one to the next, across a flat side a little + // further out than the side itself. + EXPECT_NEAR(ReachTowards(capsules[0], 0.0f, 5.0f), 1.2f, 0.05f); + EXPECT_NEAR(ReachTowards(capsules[0], 0.0f, -5.0f), 0.6f, 0.05f); + EXPECT_NEAR(ReachTowards(capsules[0], 5.0f, 0.0f), 1.0f, 0.05f); +} + +TEST(ClothEnhanced, AFittedBackLetsTheClothLieOnIt) { + std::vector capsules = {Torso()}; + cloth_body::FitTorsos(capsules, {}, Chest()); + const float home[3] = {0.0f, -3.0f, -1.5f}; + // Behind the back, where the round torso would have pushed it out to 1.65. + float lying[3] = {0.0f, -3.0f, -0.8f}; + cloth_body::PushOutOfBody(capsules, std::vector(1), home, lying); + EXPECT_NEAR(lying[2], -0.8f, 1e-6f); + // In the back, out onto it. + float inside[3] = {0.0f, -3.0f, -0.3f}; + cloth_body::PushOutOfBody(capsules, std::vector(1), home, inside); + EXPECT_NEAR(inside[2], -0.65f, 0.05f); +} + +TEST(ClothEnhanced, AHandAgainstTheTorsoDoesNotWidenIt) { + std::vector capsules = {Torso()}; + // Hanging at the torso's side, in the way of the directions either side of +x. + const cloth_body::Capsule hand = { + {2.1f, -2.0f, 0.0f}, + {2.1f, -4.0f, 0.0f}, + 0.6f + }; + cloth_body::Mesh body = Chest(); + AddBox(body, {1.8f, -4.2f, -0.8f}, {2.4f, -1.8f, 0.8f}); + cloth_body::FitTorsos(capsules, std::span(&hand, 1), body); + EXPECT_NEAR(ReachTowards(capsules[0], 5.0f, 0.0f), 1.0f, 0.05f); +} + +TEST(ClothEnhanced, ATorsoWithoutAMeshStaysRound) { + std::vector capsules = {Torso()}; + cloth_body::FitTorsos(capsules, {}, cloth_body::Mesh{}); + EXPECT_FALSE(capsules[0].fitted); + EXPECT_FLOAT_EQ(ReachTowards(capsules[0], 0.0f, -5.0f), 1.6f); +} + +} // namespace dc::test diff --git a/port/src/tests/model_gltf_test.cpp b/port/src/tests/model_gltf_test.cpp new file mode 100644 index 000000000..58a7ff5c6 --- /dev/null +++ b/port/src/tests/model_gltf_test.cpp @@ -0,0 +1,577 @@ +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../../../tools/dcdata/dcdata.hpp" +#include "data/data_read.hpp" +#include "data/model_gltf.hpp" +#include "platform/paths.hpp" +#include "platform_fixture.hpp" + +namespace dc::test { + +using namespace dc::data; + +namespace { + +// The bytes a source alias names in the raw tree: a file, the pack members inside it, or a scene +// at an offset of either ("gedit/s26/scene.scn#offset:4016"). +std::vector Original(const std::filesystem::path &raw, const std::string &alias) { + std::vector parts; + for (std::size_t start = 0;;) { + const std::size_t slash = alias.find('/', start); + parts.push_back(alias.substr(start, slash - start)); + if (slash == std::string::npos) { + break; + } + start = slash + 1; + } + std::filesystem::path file = raw; + std::size_t next = 0; + for (; next < parts.size(); next++) { + file /= parts[next].substr(0, parts[next].find('#')); + if (std::filesystem::is_regular_file(file)) { + break; + } + } + std::ifstream stream(file, std::ios::binary); + std::vector bytes((std::istreambuf_iterator(stream)), std::istreambuf_iterator()); + for (std::size_t part = next; part < parts.size(); part++) { + const std::string &name = parts[part]; + const std::size_t marker = name.find('#'); + if (part > next) { + const auto members = dcdata::ReadPack(bytes); + const std::string base = name.substr(0, marker); + std::size_t wanted = members.size(); + if (marker != std::string::npos && name.compare(marker, 7, "#entry:") == 0) { + wanted = std::stoul(name.substr(marker + 7)); + } else { + for (std::size_t m = 0; m < members.size(); m++) { + if (members[m].name == base) { + wanted = m; + break; + } + } + } + if (wanted >= members.size()) { + return {}; + } + bytes.assign(members[wanted].data.begin(), members[wanted].data.end()); + } + if (marker != std::string::npos && name.compare(marker, 8, "#offset:") == 0) { + bytes.erase(bytes.begin(), bytes.begin() + std::stoul(name.substr(marker + 8))); + } + } + return bytes; +} + +// Whether this process checks the sources of `bundle`: DC_MODEL_PART=k/parts splits a disc's bundles +// across processes run side by side, each parsing only its own. +bool InPart(const std::string &bundle) { + // DC_MODEL_ONLY narrows a run to the sources of one bundle. + if (const char *only = std::getenv("DC_MODEL_ONLY")) { + return bundle == only; + } + const std::size_t n = std::hash{}(bundle); + const char *part = std::getenv("DC_MODEL_PART"); + unsigned k = 0; + unsigned parts = 1; + if (part == nullptr || std::sscanf(part, "%u/%u", &k, &parts) != 2 || parts == 0) { + return true; + } + return n % parts == k; +} + +struct Channel { + std::int32_t type = 0; + std::vector frames; + std::vector> values; +}; + +// A MOT's channels by frame, type and target. +std::map, Channel> ReadChannels(const std::vector &bytes) { + std::map, Channel> channels; + std::size_t at = 0; + while (at + 32 <= bytes.size()) { + std::int32_t header[8]; + std::memcpy(header, bytes.data() + at, sizeof header); + Channel &channel = channels[{header[0], header[2], header[1]}]; + channel.type = header[2]; + at += 32; + for (std::int32_t k = 0; k < header[4] && at + 32 <= bytes.size(); k++, at += 32) { + std::uint32_t frame; + std::array value; + std::memcpy(&frame, bytes.data() + at, 4); + std::memcpy(value.data(), bytes.data() + at + 16, 16); + channel.frames.push_back(frame); + channel.values.push_back(value); + } + if (header[5] == 0) { + break; + } + } + return channels; +} + +// A channel's value at a frame, as the retail evaluator interpolates its keys: linearly, and a +// rotation along the arc between its normalized keys. +std::array Sample(const Channel &channel, float frame) { + const std::size_t upper = static_cast( + std::upper_bound(channel.frames.begin(), channel.frames.end(), frame, + [](float at, std::uint32_t key) { return at < float(key); }) - + channel.frames.begin()); + auto normal = [&](std::array q) { + if (channel.type != 0) { + return q; + } + const float length = std::sqrt(q[0] * q[0] + q[1] * q[1] + q[2] * q[2] + q[3] * q[3]); + for (float &v : q) { + v /= length > 1e-20f ? length : 1.0f; + } + return q; + }; + if (upper == 0) { + return normal(channel.values.front()); + } + if (upper == channel.frames.size()) { + return normal(channel.values.back()); + } + // Visibility holds its value until the next key. + if (channel.type == 50 || channel.type == 51) { + return channel.values[upper - 1]; + } + const float t = (frame - channel.frames[upper - 1]) / float(channel.frames[upper] - channel.frames[upper - 1]); + std::array a = normal(channel.values[upper - 1]); + std::array b = normal(channel.values[upper]); + if (channel.type == 0) { + float cosine = a[0] * b[0] + a[1] * b[1] + a[2] * b[2] + a[3] * b[3]; + if (cosine < 0) { + cosine = -cosine; + for (float &v : b) { + v = -v; + } + } + float from = 1 - t; + float to = t; + if (1 - cosine > 0.01f) { + const float angle = std::acos(std::min(cosine, 1.0f)); + from = std::sin((1 - t) * angle) / std::sin(angle); + to = std::sin(t * angle) / std::sin(angle); + } + return normal({from * a[0] + to * b[0], from * a[1] + to * b[1], from * a[2] + to * b[2], from * a[3] + to * b[3]}); + } + return {a[0] + (b[0] - a[0]) * t, a[1] + (b[1] - a[1]) * t, a[2] + (b[2] - a[2]) * t, a[3] + (b[3] - a[3]) * t}; +} + +float Difference(const Channel &channel, const std::array &a, const std::array &b) { + if (channel.type == 0) { + return 1 - std::abs(a[0] * b[0] + a[1] * b[1] + a[2] * b[2] + a[3] * b[3]); + } + float largest = 0; + // Translations, scales and colours use three lanes; a rotation all four. + for (std::size_t c = 0; c < (channel.type <= 2 ? 3u : 4u); c++) { + largest = std::max(largest, std::abs(a[c] - b[c]) / std::max(1.0f, std::abs(a[c]))); + } + return largest; +} + +} // namespace + +TEST(ModelGltf, EveryPublishedSourceRebuildsByteForByte) { + const char *data = std::getenv("DC_MODEL_DATA"); + const char *raw = std::getenv("DC_MODEL_RAW"); + if (data == nullptr || raw == nullptr) { + GTEST_SKIP() << "Set DC_MODEL_DATA to data converted by dcdata convert-models from the raw tree in DC_MODEL_RAW"; + } + dc::test::EnvSnapshot environment = dc::test::SaveEnv("DC_DATA"); + dc::test::SetEnv("DC_DATA", data, 1); + PathsReset(); + PathsConfigure({.app_dir = "chronicle", .env_data = "DC_DATA", .env_save = "DC_SAVE"}); + DataReadReset(); + model_gltf::Reset(); + + std::vector bytes; + ASSERT_TRUE(DataReadReadFile("models.json", bytes)); + const nlohmann::json index = nlohmann::json::parse(bytes.begin(), bytes.end()); + std::map failures; + int checked = 0; + for (const auto &[alias, entry] : index.items()) { + // Motions too: the game reads a motion's vertex channels by their place in it. + if (entry.at("kind") == "cloths" || entry.at("kind") == "children" || !InPart(entry.at("bundle"))) { + continue; + } + checked++; + const std::vector *rebuilt = model_gltf::Rebuild(alias); + std::vector original = Original(raw, alias); + std::string failure; + if (rebuilt == nullptr) { + failure = "not rebuilt"; + } else if (original.size() < rebuilt->size()) { + failure = "longer than the source"; + } else { + original.resize(rebuilt->size()); + if (original != *rebuilt) { + failure = "differs"; + } + } + if (!failure.empty() && failures[entry.at("kind").get() + " " + failure]++ < 5) { + ADD_FAILURE() << alias << ": " << failure; + } + } + EXPECT_GT(checked, 0); + for (const auto &[failure, count] : failures) { + ADD_FAILURE() << count << " sources " << failure; + } + model_gltf::Reset(); + dc::test::RestoreEnv(environment); + PathsReset(); + PathsConfigure({.app_dir = "chronicle", .env_data = "DC_DATA", .env_save = "DC_SAVE"}); + DataReadReset(); +} + +TEST(ModelGltf, EveryMotionKeepsItsPosesInEveryAction) { + const char *data = std::getenv("DC_MODEL_DATA"); + const char *raw = std::getenv("DC_MODEL_RAW"); + if (data == nullptr || raw == nullptr) { + GTEST_SKIP() << "Set DC_MODEL_DATA to data converted by dcdata convert-models from the raw tree in DC_MODEL_RAW"; + } + dc::test::EnvSnapshot environment = dc::test::SaveEnv("DC_DATA"); + dc::test::SetEnv("DC_DATA", data, 1); + PathsReset(); + PathsConfigure({.app_dir = "chronicle", .env_data = "DC_DATA", .env_save = "DC_SAVE"}); + DataReadReset(); + model_gltf::Reset(); + + std::vector bytes; + ASSERT_TRUE(DataReadReadFile("models.json", bytes)); + const nlohmann::json index = nlohmann::json::parse(bytes.begin(), bytes.end()); + std::map sidecars; + int motions = 0; + int bad = 0; + int larger = 0; + double worst = 0; + for (const auto &[alias, entry] : index.items()) { + if ((entry.at("kind") != "animations" && entry.at("kind") != "motion_channels") || !InPart(entry.at("bundle"))) { + continue; + } + motions++; + const std::string bundle = entry.at("bundle"); + if (!sidecars.contains(bundle)) { + const std::string sidecar = dcdata::PublishedPath(bundle.substr(0, bundle.size() - 5) + ".game.json", + [](std::string_view prefix) { return DataFileSize(prefix) > 0; }); + ASSERT_TRUE(DataReadReadFile(sidecar, bytes)) << sidecar; + sidecars[bundle] = nlohmann::json::parse(bytes.begin(), bytes.end()); + } + // A scene's own motion plays whole; a character's, action by action. + const bool scene = entry.at("kind") == "motion_channels"; + const nlohmann::json &resource = + scene ? sidecars[bundle] : sidecars[bundle].at("resources").at("animations").at(entry.at("index").get()); + const std::vector *rebuilt = model_gltf::Rebuild(alias); + if (rebuilt == nullptr) { + if (bad++ < 5) { + ADD_FAILURE() << alias << ": not rebuilt"; + } + continue; + } + // DC_MODEL_DUMP= keeps each rebuilt motion there, to compare with the disc's. + if (const char *dump = std::getenv("DC_MODEL_DUMP")) { + std::string name = alias; + std::replace(name.begin(), name.end(), '/', '_'); + std::ofstream(std::filesystem::path(dump) / name, std::ios::binary) + .write(reinterpret_cast(rebuilt->data()), static_cast(rebuilt->size())); + } + const auto original = ReadChannels(Original(raw, alias)); + const auto copy = ReadChannels(*rebuilt); + std::size_t original_keys = 0; + std::size_t copy_keys = 0; + for (const auto &[_, channel] : original) { + original_keys += channel.frames.size(); + } + for (const auto &[_, channel] : copy) { + copy_keys += channel.frames.size(); + } + // The game reads every vertex channel of a frame at the first one's key and blends towards + // keys by frame, so each channel keeps the disc's key frames, and none is added. + bool same_frames = true; + for (const auto &[id, channel] : original) { + const auto found = copy.find(id); + if (!channel.frames.empty() && (found == copy.end() || found->second.frames != channel.frames)) { + same_frames = false; + } + } + for (const auto &[id, channel] : copy) { + if (!channel.frames.empty() && (!original.contains(id) || original.at(id).frames.empty())) { + same_frames = false; + } + } + if (!same_frames && larger++ < 5) { + ADD_FAILURE() << alias << ": key frames differ from the disc's (" << copy_keys << " keys, the disc's " + << original_keys << ")"; + } + float largest = 0; + std::string where; + std::uint32_t last = 0; + for (const auto &[_, channel] : original) { + for (std::uint32_t frame : channel.frames) { + last = std::max(last, frame); + } + } + const nlohmann::json actions = + scene ? nlohmann::json::array({ + {{"start", 0}, {"end", last}} + }) + : resource.at("actions"); + for (const auto &[id, channel] : original) { + if (channel.frames.empty()) { + continue; + } + const auto found = copy.find(id); + for (const nlohmann::json &action : actions) { + for (std::uint32_t f = action.at("start"); f <= action.at("end").get(); f++) { + float difference = 1; + if (found != copy.end() && !found->second.frames.empty()) { + difference = Difference(channel, Sample(channel, f), Sample(found->second, f)); + } + if (difference > largest) { + largest = difference; + where = std::format("frame {} type {} target {} at {}", id[0], id[1], id[2], f); + } + } + } + } + worst = std::max(worst, double(largest)); + if (largest > 1e-3f && bad++ < 10) { + ADD_FAILURE() << alias << ": " << largest << " at " << where; + } + } + EXPECT_GT(motions, 0); + EXPECT_EQ(bad, 0) << "of " << motions << " motions"; + EXPECT_EQ(larger, 0) << "motions rebuilt with other key frames than the disc's"; + std::printf("%d motions, largest difference %g\n", motions, worst); + model_gltf::Reset(); + dc::test::RestoreEnv(environment); + PathsReset(); + PathsConfigure({.app_dir = "chronicle", .env_data = "DC_DATA", .env_save = "DC_SAVE"}); + DataReadReset(); +} + +TEST(ModelGltf, SplicingPutsAnEmptiedPackMemberBack) { + const char *data = std::getenv("DC_MODEL_DATA"); + const char *raw = std::getenv("DC_MODEL_RAW"); + if (data == nullptr || raw == nullptr) { + GTEST_SKIP() << "Set DC_MODEL_DATA to data converted by dcdata convert-models from the raw tree in DC_MODEL_RAW"; + } + dc::test::EnvSnapshot environment = dc::test::SaveEnv("DC_DATA"); + dc::test::SetEnv("DC_DATA", data, 1); + PathsReset(); + PathsConfigure({.app_dir = "chronicle", .env_data = "DC_DATA", .env_save = "DC_SAVE"}); + DataReadReset(); + + // Toan's pack with his model and weights emptied, as stripping the converted data leaves them. + std::vector original = Original(raw, "chara/c01d.chr"); + ASSERT_FALSE(original.empty()); + std::vector stripped = original; + for (const char *name : {"c01d.mds", "c01dbas.wgt", "c01dbas.bbp"}) { + std::size_t at = 0; + while (at + 76 <= stripped.size() && std::strcmp(reinterpret_cast(stripped.data() + at), name) != 0) { + std::uint32_t next; + std::memcpy(&next, stripped.data() + at + 72, 4); + at += next; + } + ASSERT_LT(at + 76, stripped.size()) << name; + std::uint32_t offset; + std::uint32_t size; + std::memcpy(&offset, stripped.data() + at + 64, 4); + std::memcpy(&size, stripped.data() + at + 68, 4); + std::fill_n(stripped.begin() + at + offset, size, 0); + std::fill_n(stripped.begin() + at + 68, 4, 0); + } + model_gltf::Splice("chara/c01d.chr", stripped); + const auto members = dcdata::ReadPack(stripped); + for (const auto &member : dcdata::ReadPack(original)) { + if (member.name == "c01d.mds" || member.name.ends_with(".wgt") || member.name.ends_with(".bbp")) { + const auto found = std::find_if(members.begin(), members.end(), + [&](const auto &other) { return other.name == member.name; }); + ASSERT_NE(found, members.end()); + EXPECT_TRUE(found->data == member.data) << member.name; + } + } + model_gltf::Reset(); + dc::test::RestoreEnv(environment); + PathsReset(); + PathsConfigure({.app_dir = "chronicle", .env_data = "DC_DATA", .env_save = "DC_SAVE"}); + DataReadReset(); +} + +namespace { + +// The byte ranges of the motion members inside a pack file, by walking its packs as the game does. +void MotionRanges(std::span bytes, std::size_t base, std::vector> &out, + int depth = 0) { + std::size_t at = 0; + while (depth < 6 && at + 76 <= bytes.size() && bytes[at] != 0) { + std::uint32_t offset; + std::uint32_t size; + std::uint32_t next; + std::memcpy(&offset, bytes.data() + at + 64, 4); + std::memcpy(&size, bytes.data() + at + 68, 4); + std::memcpy(&next, bytes.data() + at + 72, 4); + if (offset < 76 || next < 76 || at + offset + size > bytes.size() || std::memchr(bytes.data() + at, 0, 64) == nullptr) { + return; + } + const std::string_view name(reinterpret_cast(bytes.data() + at)); + if (name.ends_with(".mot") || name.ends_with(".MOT")) { + out.emplace_back(base + at + 68, base + at + 72); + out.emplace_back(base + at + offset, base + at + next); + } else { + MotionRanges(bytes.subspan(at + offset, size), base + at + offset, out, depth + 1); + } + at += next; + } +} + +} // namespace + +// The data with its converted sources emptied (dcdata strip-models) reads back as the disc: every +// file, spliced from the published models as the game reads it, equals the disc's, but for the +// motions, which come back from their actions and are compared pose by pose above. +// The game reads every vertex channel of a motion at its first channel's key, so a motion's channels +// must stay in the disc's order: rebuilt in another, Saget's vertex animation in the attract movie wrote +// past his mesh and crashed. Every motion inside a pack comes back as the disc has it. +TEST(ModelGltf, MotionsGoInOnlyAsTheDiscHasThem) { + const char *data = std::getenv("DC_MODEL_DATA"); + const char *raw = std::getenv("DC_MODEL_RAW"); + if (data == nullptr || raw == nullptr) { + GTEST_SKIP() << "Set DC_MODEL_DATA to data converted by dcdata convert-models from the raw tree in DC_MODEL_RAW"; + } + dc::test::EnvSnapshot environment = dc::test::SaveEnv("DC_DATA"); + dc::test::SetEnv("DC_DATA", data, 1); + PathsReset(); + PathsConfigure({.app_dir = "chronicle", .env_data = "DC_DATA", .env_save = "DC_SAVE"}); + DataReadReset(); + model_gltf::Reset(); + + std::vector bytes; + ASSERT_TRUE(DataReadReadFile("models.json", bytes)); + const nlohmann::json index = nlohmann::json::parse(bytes.begin(), bytes.end()); + std::set files; + for (const auto &[alias, entry] : index.items()) { + if (entry.at("kind") != "animations" && entry.at("kind") != "motion_channels") { + continue; + } + std::string file; + std::size_t start = 0; + for (;;) { + const std::size_t slash = alias.find('/', start); + const std::string part = alias.substr(start, slash - start); + file += (file.empty() ? "" : "/") + part.substr(0, part.find('#')); + if (std::filesystem::is_regular_file(std::filesystem::path(raw) / file) || slash == std::string::npos) { + break; + } + start = slash + 1; + } + if (file != alias && InPart(file)) { + files.insert(file); + } + } + int differing = 0; + for (const std::string &file : files) { + std::vector read; + ASSERT_TRUE(DataReadReadFile(file, read)) << file; + model_gltf::Splice(file, read); + const std::vector original = Original(raw, file); + std::vector> motions; + MotionRanges(original, 0, motions); + for (const auto &[first, last] : motions) { + if (last > read.size() || !std::equal(read.begin() + first, read.begin() + last, original.begin() + first)) { + if (differing++ < 10) { + ADD_FAILURE() << file << ": a motion differs from the disc's at byte " << first; + } + break; + } + } + } + EXPECT_GT(files.size(), 0u); + EXPECT_EQ(differing, 0) << "of " << files.size() << " files"; + model_gltf::Reset(); + dc::test::RestoreEnv(environment); + PathsReset(); + PathsConfigure({.app_dir = "chronicle", .env_data = "DC_DATA", .env_save = "DC_SAVE"}); + DataReadReset(); +} + +TEST(ModelGltf, StrippedDataReadsBackAsTheDisc) { + const char *data = std::getenv("DC_MODEL_STRIPPED"); + const char *raw = std::getenv("DC_MODEL_RAW"); + if (data == nullptr || raw == nullptr) { + GTEST_SKIP() << "Set DC_MODEL_STRIPPED to data stripped by dcdata strip-models, and DC_MODEL_RAW to the disc's"; + } + dc::test::EnvSnapshot environment = dc::test::SaveEnv("DC_DATA"); + dc::test::SetEnv("DC_DATA", data, 1); + PathsReset(); + PathsConfigure({.app_dir = "chronicle", .env_data = "DC_DATA", .env_save = "DC_SAVE"}); + DataReadReset(); + + std::vector bytes; + ASSERT_TRUE(DataReadReadFile("models.json", bytes)); + const nlohmann::json index = nlohmann::json::parse(bytes.begin(), bytes.end()); + std::set files; + for (const auto &[alias, entry] : index.items()) { + std::string file; + std::size_t start = 0; + for (;;) { + const std::size_t slash = alias.find('/', start); + const std::string part = alias.substr(start, slash - start); + file += (file.empty() ? "" : "/") + part.substr(0, part.find('#')); + if (std::filesystem::is_regular_file(std::filesystem::path(raw) / file) || slash == std::string::npos) { + break; + } + start = slash + 1; + } + if (InPart(file)) { + files.insert(file); + } + } + int differing = 0; + for (const std::string &file : files) { + std::vector read; + ASSERT_TRUE(DataReadReadFile(file, read)) << file; + model_gltf::Splice(file, read); + // Motions too: they come back as the disc has them, channel order and all. + const std::vector original = Original(raw, file); + std::size_t first = original.size(); + for (std::size_t at = 0; at < std::min(original.size(), read.size()); at++) { + if (read[at] != original[at]) { + first = at; + break; + } + } + if ((first < original.size() || read.size() != original.size()) && differing++ < 10) { + ADD_FAILURE() << file << ": differs from the disc at byte " << first; + } + } + EXPECT_EQ(differing, 0) << "of " << files.size() << " files"; + model_gltf::Reset(); + dc::test::RestoreEnv(environment); + PathsReset(); + PathsConfigure({.app_dir = "chronicle", .env_data = "DC_DATA", .env_save = "DC_SAVE"}); + DataReadReset(); +} + +} // namespace dc::test diff --git a/tools/dcdata/character_profile.hpp b/tools/dcdata/character_profile.hpp new file mode 100644 index 000000000..110fc9b45 --- /dev/null +++ b/tools/dcdata/character_profile.hpp @@ -0,0 +1,407 @@ +#pragma once + +// Reads Dark Cloud's character configurations (.cfg) and cloth files (.clo) into the typed commands +// Kazarov's model converter takes, word for word as the game's own readers do: ReadInfo +// (chararead.cpp), CTextureAnime::LoadCFGFile (textureanime.cpp) and InitCloth (clothread.cpp). +// Header-only, like dcdata.hpp, so the port's tests can use it. It includes no game header. + +#include +#include +#include +#include +#include +#include +#include + +namespace dcdata::profile { + +enum class Arg { + String, + Integer, + Float +}; + +struct CommandSpec { + std::string_view name; + std::vector args; +}; + +// ReadInfo's command table, in its order. +inline const std::vector &CharacterCommands() { + using enum Arg; + static const std::vector commands = { + {"VERTEX_ANIME", {Integer} }, + {"SHADOW_VERTEX_ANIME", {Integer} }, + {"MODEL", {String} }, + {"SHADOW_MODEL", {String} }, + {"MOTION", {Integer, String, String, String}}, + {"SHADOW_MOTION", {String, String, String} }, + {"KEY", {Integer, Integer, Float} }, + {"KEY_START", {Integer} }, + {"MOTION_END", {} }, + {"CLOTH", {String} }, + {"BODY_SIZE", {Float, Float, Float} }, + {"ALLOC_MDT", {String} }, + {"ALLOC_DBUFF", {String} }, + {"ALLOC_SHADOW_MDT", {String} }, + {"ALLOC_SHADOW_DBUFF", {String} }, + {"IMG", {Integer, String} }, + {"IMG_END", {} }, + {"FOOT", {Float, Float} }, + {"EVENT", {Float, Integer, Integer} }, + }; + return commands; +} + +// CTextureAnime::LoadCFGFile's tags, read from the same file by the script interpreter. +inline const std::vector &TextureAnimeCommands() { + using enum Arg; + static const std::vector commands = { + {"TEX_ANIME", {Integer, Integer} }, + {"TEX_ANIME_DATA", {String, Integer, Integer, Integer, Integer, String, Integer, Integer, Integer, Integer} }, + {"TEX_ANIME_DATA2", + {String, Integer, Integer, Integer, Integer, String, Integer, Integer, Integer, Integer, Integer, Integer, + Integer, Integer} }, + {"TEX_SCROLL_DATA", {String, Integer, Integer, Integer, Integer, String, Integer, Integer, Float, Float, Integer, Integer}}, + {"TEX_ANIME_END", {} }, + }; + return commands; +} + +// InitCloth's command table. +inline const std::vector &ClothCommands() { + using enum Arg; + static const std::vector commands = { + {"SIZE", {Integer, Integer} }, + {"FRAME", {String} }, + {"NORMAL", {Float} }, + {"FOLLOW", {Float, Float, Float} }, + {"K", {Float, Float, Float} }, + {"WINDEFFECT", {Float} }, + {"GRAVITY", {Float, Float, Float} }, + {"POLYDIV", {String} }, + {"BOUND", {String, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float, Float}}, + }; + return commands; +} + +inline bool IsSpace(char c) { + return c == ' ' || c == '\t' || c == '\n' || c == '\r'; +} + +// Blanks // and /* */ comments as the game's PreProcess does before reading a configuration. +inline std::string PreProcess(std::string_view raw) { + std::string text(raw); + const std::size_t size = text.size(); + auto at = [&](std::size_t pos) { return pos < size ? text[pos] : '\0'; }; + std::size_t pos = 0; + while (pos < size) { + if (at(pos) == '/' && at(pos + 1) == '/') { + while (pos < size && text[pos] != '\n' && text[pos] != '\r') { + text[pos++] = ' '; + } + } + if (at(pos) == '/' && at(pos + 1) == '*') { + while (pos < size) { + if (at(pos) == '*' && at(pos + 1) == '/') { + text[pos] = ' '; + text[pos + 1] = ' '; + break; + } + text[pos++] = ' '; + } + continue; + } + pos++; + } + return text; +} + +// Shift-JIS leaves ASCII bytes as they are only outside double-byte characters, so a comment is +// walked character by character from its start. +inline std::vector SjisCharacters(std::string_view text) { + std::vector starts; + for (std::size_t i = 0; i < text.size();) { + starts.push_back(i); + const auto lead = static_cast(text[i]); + i += (lead >= 0x81 && lead <= 0x9F) || (lead >= 0xE0 && lead <= 0xFC) ? 2 : 1; + } + return starts; +} + +// A space, a tab or an ideographic space. +inline bool IsBlankAt(std::string_view text, std::size_t at) { + return text[at] == ' ' || text[at] == '\t' || text.substr(at, 2) == "\x81\x40"; +} + +// An ASCII or a full-width digit. +inline bool IsDigitAt(std::string_view text, std::size_t at) { + const auto a = static_cast(text[at]); + const auto b = at + 1 < text.size() ? static_cast(text[at + 1]) : 0; + return (a >= '0' && a <= '9') || (a == 0x82 && b >= 0x4F && b <= 0x58); +} + +// The name a KEY line's comment gives its action: the comment's text without the blanks around it +// and the action's index after it, as in `KEY 10, 20, 0.05 //立ち 0`. Empty without a comment, or +// with one that is only the index. +inline std::string KeyComment(std::string_view raw, std::size_t command_pos) { + const std::size_t line_end = std::min(raw.find_first_of("\r\n", command_pos), raw.size()); + const std::string_view line = raw.substr(command_pos, line_end - command_pos); + const std::size_t slash = line.find("//"); + if (slash == std::string_view::npos) { + return {}; + } + const std::string_view comment = line.substr(slash + 2); + const std::vector chars = SjisCharacters(comment); + std::size_t first = 0; + std::size_t end = chars.size(); + while (first < end && IsBlankAt(comment, chars[first])) { + first++; + } + while (end > first && IsBlankAt(comment, chars[end - 1])) { + end--; + } + std::size_t digits = end; + while (digits > first && IsDigitAt(comment, chars[digits - 1])) { + digits--; + } + // A comment that is only the index names nothing. + if (digits < end && (digits == first || IsBlankAt(comment, chars[digits - 1]))) { + end = digits; + while (end > first && IsBlankAt(comment, chars[end - 1])) { + end--; + } + } + if (first == end) { + return {}; + } + const std::size_t stop = end < chars.size() ? chars[end] : comment.size(); + return std::string(comment.substr(chars[first], stop - chars[first])); +} + +// One typed command, as Kazarov's converter reads it. Strings go as their raw bytes, in hex; the +// converter decodes them from Shift-JIS as it does Dark Cloud 2's. +class CommandJson { +public: + explicit CommandJson(std::string_view name) : out_(std::format(R"({{"command":"{}","args":[)", name)) {} + + void String(std::string_view bytes) { + constexpr char kHex[] = "0123456789abcdef"; + std::string hex; + for (unsigned char c : bytes) { + hex += kHex[c >> 4]; + hex += kHex[c & 0xF]; + } + Arg(std::format(R"({{"type":"string","bytes":"{}"}})", hex)); + } + + void Integer(int value) { Arg(std::format(R"({{"type":"int","value":{}}})", value)); } + + void Float(float value) { Arg(std::format(R"({{"type":"float","value":{}}})", value)); } + + void None() { Arg(R"({"type":"none"})"); } + + void Invalid() { Arg(R"({"type":"invalid","value":null})"); } + + std::string Finish() const { return out_ + "]}"; } + +private: + void Arg(std::string_view json) { + out_ += args_++ ? "," : ""; + out_ += json; + } + + std::string out_; + int args_ = 0; +}; + +// One command a reader found, at the offset of its keyword, with its typed arguments as JSON. +struct Found { + std::size_t pos; + std::string json; +}; + +// The dialects of the game's two word readers: ReadInfo's and InitCloth's (chararead.cpp, +// clothread.cpp), and the script interpreter's (scriptinterpreter.cpp). +enum class Dialect { + Character, + Cloth, + Interpreter +}; + +class WordReader { +public: + WordReader(std::string_view text, Dialect dialect) : text_(text), dialect_(dialect) {} + + bool SkipSpace() { + while (pos_ < text_.size() && IsSpace(text_[pos_])) { + pos_++; + } + return pos_ < text_.size(); + } + + // The next command word, or false at the end of the file. + bool Word(std::string &word, std::size_t &start) { + if (!SkipSpace()) { + return false; + } + start = pos_; + word.clear(); + while (pos_ < text_.size() && !IsSpace(text_[pos_])) { + word += text_[pos_++]; + } + // ReadInfo takes a word cut short by the file's end; the others read nothing. + if (pos_ == text_.size() && dialect_ != Dialect::Character) { + return false; + } + pos_ = std::min(pos_ + 1, text_.size()); + return true; + } + + // Reads a command's arguments; false at the end of the file, where the game runs nothing. + bool Args(const std::vector &types, CommandJson &json) { + if (types.empty()) { + return true; + } + if (!SkipSpace()) { + return false; + } + for (Arg type : types) { + if (!SkipSpace()) { + return false; + } + std::string token; + if (dialect_ == Dialect::Interpreter && text_[pos_] == ',') { + pos_++; + if (!SkipSpace()) { + return false; + } + } + for (;;) { + if (pos_ >= text_.size()) { + return false; + } + const char c = text_[pos_++]; + if (c == ',' || IsSpace(c)) { + break; + } + token += c; + } + if (!ReadArg(type, token, json)) { + json.Invalid(); + return true; + } + } + return true; + } + +private: + bool ReadArg(Arg type, const std::string &token, CommandJson &json) const { + const bool interpreter = dialect_ == Dialect::Interpreter; + switch (type) { + case Arg::String: { + if (token.empty() || token[0] != '"') { + return false; + } + const std::size_t close = token.find('"', 1); + if (close == std::string::npos) { + return false; + } + json.String(std::string_view(token).substr(1, close - 1)); + return true; + } + case Arg::Integer: + for (char c : token) { + if ((c < '0' || c > '9') && !(interpreter && c == '-')) { + return false; + } + } + json.Integer(std::atoi(token.c_str())); + return true; + case Arg::Float: + for (char c : token) { + if ((c < '0' || c > '9') && c != '.' && c != '-') { + return false; + } + } + json.Float(static_cast(std::atof(token.c_str()))); + return true; + } + return false; + } + + std::string_view text_; + Dialect dialect_; + std::size_t pos_ = 0; +}; + +// Reads every command of one table from a file, as the game's reader of that table would. +inline std::vector ReadCommands(std::string_view raw, std::string_view text, + const std::vector &table, Dialect dialect) { + std::vector found; + WordReader reader(text, dialect); + std::string word; + std::size_t start = 0; + while (reader.Word(word, start)) { + const auto spec = std::find_if(table.begin(), table.end(), + [&](const CommandSpec &command) { return command.name == word; }); + if (spec == table.end()) { + continue; + } + CommandJson json(spec->name); + // KEY carries its comment's name first, as Dark Cloud 2's KEY does: KEY name, start, end, step. + if (spec->name == "KEY") { + const std::string name = KeyComment(raw, start); + if (name.empty()) { + json.None(); + } else { + json.String(name); + } + } + if (!reader.Args(spec->args, json)) { + break; + } + found.push_back({start, json.Finish()}); + } + return found; +} + +inline bool EndsWithFolded(std::string_view text, std::string_view suffix) { + return text.size() >= suffix.size() && + std::equal(suffix.begin(), suffix.end(), text.end() - suffix.size(), [](char a, char b) { + return (a >= 'A' && a <= 'Z' ? a - 'A' + 'a' : a) == b; + }); +} + +// The typed commands of a character configuration or cloth file, as JSON, or "null" for a file the +// game doesn't read as one: a .cfg with neither MODEL nor MOTION, or any other kind of file. +inline std::string Read(std::string_view alias, std::string_view data) { + std::vector found; + if (EndsWithFolded(alias, ".clo")) { + found = ReadCommands(data, data, ClothCommands(), Dialect::Cloth); + } else if (EndsWithFolded(alias, ".cfg")) { + const std::string text = PreProcess(data); + found = ReadCommands(data, text, CharacterCommands(), Dialect::Character); + const bool character = std::any_of(found.begin(), found.end(), [](const Found &command) { + return command.json.starts_with(R"({"command":"MODEL")") || + command.json.starts_with(R"({"command":"MOTION")"); + }); + if (!character) { + return "null"; + } + std::vector textures = ReadCommands(data, text, TextureAnimeCommands(), Dialect::Interpreter); + found.insert(found.end(), textures.begin(), textures.end()); + std::stable_sort(found.begin(), found.end(), + [](const Found &a, const Found &b) { return a.pos < b.pos; }); + } else { + return "null"; + } + std::string out = R"({"schema_version":1,"commands":[)"; + for (std::size_t i = 0; i < found.size(); i++) { + out += i ? "," : ""; + out += found[i].json; + } + return out + "]}"; +} + +} // namespace dcdata::profile diff --git a/tools/dcdata/dcdata.hpp b/tools/dcdata/dcdata.hpp index 198bc362e..5da7fc7f1 100644 --- a/tools/dcdata/dcdata.hpp +++ b/tools/dcdata/dcdata.hpp @@ -75,6 +75,24 @@ inline std::string FoldPath(std::string_view path) { return out; } +// Where a published model bundle's file sits in the data. Bundles are named after their source, so +// a pack's members land in a folder named like the pack ("chara/fishing.pak/uki.gltf"); the data +// keeps the pack itself under that name, so such a folder takes ".assets" after it. +inline std::string PublishedPath(std::string_view path, const std::function &is_file) { + std::string out; + std::size_t start = 0; + for (std::size_t slash = path.find('/'); slash != std::string_view::npos; slash = path.find('/', start)) { + out.append(path.substr(start, slash - start)); + if (is_file(out)) { + out += ".assets"; + } + out += '/'; + start = slash + 1; + } + out.append(path.substr(start)); + return out; +} + inline bool IsSafeRelative(std::string_view path) { if (path.empty() || path.back() == '/') { return false; diff --git a/tools/dcdata/main.cpp b/tools/dcdata/main.cpp index 99f8e9688..7d33d489a 100644 --- a/tools/dcdata/main.cpp +++ b/tools/dcdata/main.cpp @@ -1,5 +1,7 @@ #include "kazarov/extract_job.hpp" +#include "kazarov/profile_reader.hpp" #include "kazarov_bridge/rust/host_jobs.h" +#include "kazarov_bridge/rust/host_models.h" #include "kazarov_bridge/rust/lib.h" #include @@ -19,6 +21,7 @@ #include +#include "character_profile.hpp" #include "dcdata.hpp" #include "sound_data.hpp" @@ -177,6 +180,63 @@ void BakeTextures(const fs::path &raw, const fs::path &data) { dcdata::Log(stdout, "baked {} texture sources into {} PNGs\n", textures.sources, textures.pngs); } +// Converts the raw tree's textures and models into `output`, the models as glTF with Dark Cloud's +// character configurations read as the game reads them. +void ConvertInto(const fs::path &raw, const fs::path &output) { + auto reader = kazarov::make_profile_reader( + [](std::string_view alias, std::string_view data) { return dcdata::profile::Read(alias, data); }); + const kazarov::ModelConvertReport report = + kazarov::models_convert_dc1(Utf8Path(raw), Utf8Path(output), std::move(reader), WorkerCount()); + dcdata::Log(stdout, "converted {} texture sources into {} PNGs and published {} model files\n", + report.texture_sources, report.pngs, report.files); + if (report.unresolved_motions != 0) { + dcdata::Log(stdout, "{} motions fit no model\n", report.unresolved_motions); + } +} + +void ConvertModels(const fs::path &raw, const fs::path &output) { + if (fs::exists(output)) { + throw std::runtime_error("model output already exists: " + output.string()); + } + fs::create_directories(output); + ConvertInto(raw, output); +} + +bool IsModelAsset(const fs::path &path) { + const std::string name = path.filename().string(); + return name.ends_with(".gltf") || name.ends_with(".gltf.bin") || name.ends_with(".game.json"); +} + +// Copies the converted models and their index into the data, replacing an earlier conversion's. +// The disc's own model files stay beside them; the port rebuilds them from these as it reads them. +void MergeModelAssets(const fs::path &stage, const fs::path &data) { + const fs::path staged = stage / "data"; + std::size_t copied = 0; + for (fs::recursive_directory_iterator it(staged), end; it != end; ++it) { + const fs::path relative = it->path().lexically_relative(staged); + if (!it->is_regular_file() || *relative.begin() == "textures" || !IsModelAsset(relative)) { + continue; + } + const auto utf8 = relative.generic_u8string(); + const fs::path target = data / dcdata::PublishedPath(std::string(utf8.begin(), utf8.end()), [&](std::string_view prefix) { + return fs::exists(data / prefix) && !fs::is_directory(data / prefix); + }); + fs::create_directories(target.parent_path()); + fs::copy_file(it->path(), target, fs::copy_options::overwrite_existing); + copied++; + } + fs::copy_file(staged / "models.json", data / "models.json", fs::copy_options::overwrite_existing); + dcdata::Log(stdout, "published {} model files\n", copied); +} + +// Converts the textures and models in one pass, since the models' glTF reference the textures' PNGs. +void BakeAssets(const fs::path &raw, const fs::path &data) { + TextureStage stage; + ConvertInto(raw, stage.root); + MergeTextureAssets(stage.root, data); + MergeModelAssets(stage.root, data); +} + rust::Slice Bytes(const dcdata::PackMember &member) { return {member.data.data(), member.data.size()}; } @@ -321,6 +381,9 @@ int Usage() { " dcdata bake-textures \n" " dcdata publish-sound \n" " dcdata texture-map \n" + " dcdata convert-models \n" + " dcdata bake-assets \n" + " dcdata strip-models \n" " dcdata pack [output.zip]\n" " dcdata list \n", stderr); @@ -403,7 +466,7 @@ int Extract(const std::filesystem::path &source, const std::filesystem::path &ou if (result != 0) { return result; } - BakeTextures(out, out); + BakeAssets(out, out); return 0; } @@ -428,6 +491,20 @@ int main(int argc, char **argv) { BakeTextures(argv[2], argv[3]); return 0; } + if (argc == 3 && std::strcmp(argv[1], "strip-models") == 0) { + // The game rebuilds what this empties from the published models as it reads it. + const std::uint64_t stripped = kazarov::models_strip_dc1(Utf8Path(argv[2])); + dcdata::Log(stdout, "emptied {} converted sources; the game rebuilds them from their glTF\n", stripped); + return 0; + } + if (argc == 4 && std::strcmp(argv[1], "bake-assets") == 0) { + BakeAssets(argv[2], argv[3]); + return 0; + } + if (argc == 4 && std::strcmp(argv[1], "convert-models") == 0) { + ConvertModels(argv[2], argv[3]); + return 0; + } if (argc == 4 && std::strcmp(argv[1], "texture-map") == 0) { WriteTextureMap(argv[2], argv[3]); return 0; diff --git a/tools/kazarov b/tools/kazarov index 223d1e4dd..0dcfcdaa9 160000 --- a/tools/kazarov +++ b/tools/kazarov @@ -1 +1 @@ -Subproject commit 223d1e4dd3db34af85faa7e15277b615484e62ae +Subproject commit 0dcfcdaa9086bb5c49cadf944dfb92cab80fd8bc