- Generated on Mon Jun 2 2025 16:48:51 for MT Core (C++) by
1.13.2
|
MT Core (C++)
Core library for replacing C++ standard in project usage
|
https://code.matthewtolman.dev/mtolman/mtcore
This is my personal "standard" or "core" library for personal projects. It includes custom code which I've written, and it also vendors several third party libraries in the vendor directory (e.g. raylib, SQLite, Clay, etc.).
The goal is for this project to work with WASM (except for threads and some vendor libraries), Windows (not tested yet), MacOS (tested), and Linux (tested on PopOS! and Fedora).
The project is split into two main categories: core, and thread.
The project is licensed under the Apache 2.0 License, including MTCore and MTTest. 3rd party libraries are provided in the vendors folder underneath their appropriate licenses. Only one is a direct optional dependency, cpptrace, which is licensed under the MIT license. If you don't want to depend on cpptrace, set the CMake variable COMPILE_WITH_STACKTRACES to OFF before compiling. This will turn off all code which relies on cpptrace (which also turns off mtcore's ability to print stacktraces).
List of all vendored/distributed 3rd party code and their licenses:
For compiling and developing, install clang-format, cppcheck, cmake-format, a C++ compiler, and cmake
For documentation generation, install flex, bison, graphviz, and doxygen
You'll also need to install X11 and OpenGL files for Linux:
For emscripten support, install the EMSDK under your home directory following their instructions.
Note: WASM builds don't support stack traces or threads. The entire mtcore::thread namespace will be unavailable for WASM builds. Additionally, the cpptrace vendored library will not be available, and the "print stacktrace" macros will not print a stacktrace (just whatever message you give them).
The goal of core is to provide data structures and patterns that are reusable. A lot of how the patterns work are inspired by Zig. This includes vtables, Allocators, init/deinit, defer, asserts-in-release, errors as values, optionals, and nullable iterators.
This library is NOT thread safe!
VTables are more explicit than inheritance, and they allow for values to be allocated on the stack and passed by copy or reference while maintaining polymorphism, rather than the weird "needing a pointer/reference only" bugs with inheritance. Additionally, there is no need of virtual functions, and it makes creating new implementations (or even modifying provided implementations) just so much easier. I just find explicit vtables easier to work with and less bug-prone than inheritance, so I use VTables instead.
Additionally, VTables have the advantage of being able to read the code to know what's going on and the performance implications, rather than having it hidden away by a "compiler implementation" which can cause all sorts of issues when trying to optimize code away.
VTables are all defined ad-hoc as function pointers that simply receive some state and arguments. Nothing too crazy, and surprisingly effective.
The goal of allocators is to make it a lot easier to change allocation patterns than doing the C++ overriding new stuff. It also allows changing allocation patterns fluidly depending on what part of the code is being run. Also, by passing allocators it allows documenting which parts of the code does allocation.
Allocators do have issues when trying to use in conjunction with some C++ standard-library stuff - but that's because the C++ standard doesn't use the allocator ideas. It's doable, but not pleasant. Because of that, I'm reimplementing a lot of data structures and algorithms as needed to work better with my allocators.
Currently, my debug allocator just tracks total bytes allocated. At some point I'd like it to track stack traces, but most C++ compilers I've used don't really support the C++ stacktrace methods, and so I'll need to come up with something elee.
Constructors are awful when something might fail. When it might fail, you now have exceptions when building something. Additionally, those exceptions happen anytime something is default constructed - such as in an empty vector. If there's enough memory, then creating an empty vector should not fail - but in C++ it can. Not only that, but if the constructor is slow, then creating an empty vector will be very slow.
The reason for this is C++ constructors conflate memory allocation and value initialization and combines both into one operation - always. Really, they're two separate and distinct operations. Allocating memory is one step (which may fail), and should be customizable (i.e. choose the allocation strategy). Value initialization is a separate step (which may fail) with different requirements and timing - in the case of data structures this should happen much later (i.e. when an item is actually inserted). However, constructors force both operations to happen at the same time. Plus, once you get one constructor in C++, then you enter the ever-growing rule-of-3 rule-of-5 rule-of-6. The fact that the "rule-of-x" is constantly growing with newer C++ versions is terrifying - code that once worked with known, predictable behavior in one version could break in another version (or at least be incompatible with performance improvements - like move semantics). By not having a constructor, classes are more future-proof, easier to use, and are also easier to both allocate and initialize.
Destructors (especially with inheritance) are worse. First, if anyone ever is going to inherit from your class, then the destructor must be virtual, otherwise you get a memory leak (fun!). This immediately means a polymorphism-enabling-data-structure (e.g. vtable) appears for that class immediately. Even if you don't actually extend the class (hopefully some compilers are smart enough to optimize it away, but C++ is complicated, and it probably can't in every scenario).
Not only that, but destructors are called on copy. Including with copy constructors. That's right. Anytime you do auto a = Foo() and the compiler chooses to call Foo(const Foo&) rather than Foo(), you get a destructor call. This "fun" feature actually caused issues with my allocators implementation where I'd incorrectly report a memory leak because on some compilers I got a copy constructor call not a move constructor call.
Once anything starts getting done in destructors, you're obligated to now think of every assignment operator and how it should work. Are copies legal? If so, how do I ensure all the copy memory is always valid? Does that mean a deep copy every time? If they aren't valid, does that make my code super hard to use? What about moves? Since moves still call the destructor on the old value, how do I ensure that the move constructor's destructor call doesn't crash? How do I ensure the old memory is cleaned up? How do I make sure every field is properly handled? Should I just put everything on the heap and delete my constructors/destructors/assignment operators to avoid worrying about this? Will deleting those operators break once the "rule-of-x" grows again (spoiler: yes)?
By removing constructors and destructors (at least on anything user facing) we immediately remove the above issues. Instead, we have one operation for memory allocation, we then use init() to initialize, we use our value (copying when needed), call deinit() to un-initialize, and then clean up the memory.
This poses a question: how do we manage our deinit() calls? The answer is defer.
Defer delays execution of a statement until the end of a scope. It's basically an on-demand RAII, but with auto-captures and not needing to define a class every time you use it. I provide a defer implementation called mtdefer (I didn't use defer since I got a macro collision with a function name in flecs). Under the hood, it's basically a lambda capture that gets put into a destructor and called - it's the most straightfoward way to implement defer in C++. Ideally it would be built into the language, but then we wouldn't be using C++ (we'd be using Zig, Odin, C3, D, etc.). If you are really interested in this library, then you aren't using one of those languages, so this is pretty close. My defer does requrie a block and semicolon to come after the mtdefer macro (e.g. mtdefer { /* your code */ };), but that's due to how I implemented it. There are other defer implementations which let you use parenthesis for a single statement (e.g. defer(delete obj)), but I chose my pattern since I like being able to have longer defer blocks that clean multiple things up at once.
Asserts are generally very common in C++ code, and are used in debug builds all the time to make sure invariants are kept. This is great since it catches bugs in debug builds. The problem? These are removed in release builds - which means that all those "checks" we had no longer exist! This highlights a fundamental flaw in the mentality of asserts - asserts assume that all bugs and invariant violations will be caught before delivery to the customer - a statement that any customer bug report quickly disproves. Having a debug assertion that we aren't accessing out of bounds doesn't help when a hacker forces an out-of-bounds check in production. So, what's the solution? Simple, keep the assertions.
To do this, I introduced ensure which is an assertion that stays in release builds. It also prints the file line where the assertion fails, and errno information as well (mostly since I work with C APIs, and it's useful to see the errno information). This ensures that if something goes terribly wrong (i.e. an invariant is broken), then my program will crash rather than try to operate in an illegal state (e.g. write an instruction address to the 1000th element of a 10 element array - something typical of Remote Code Execution vulnerabilities).
For those wondering what happens when the program crashes, the answer is simple: restart it. Either the program is a desktop or mobile application, at which point the user will simply get frustrated and then restart (an experience they're already familiar with); a browser WASM application (at which point a page refresh fixes it); or a server application which should be managed by a system manager (e.g. systemd) which will automatically restart the program (or at the very least have a down detector that restarts the whole server). In any case, it's a much smaller deal to crash and restart the program then it is to operate in an illegal state and have to deal with a security vulnerability. Plus, it does make fuzzy testing release builds much more effective.
I'm not a big fan of the whole "throw an exception and catch it somewhere on the stack maybe" way of doing errors. It's very bug-prone and has caused so many headaches in every programming language I've used. It's not a C++ issue, it's just a fundamentally broken way of doing things. The biggest flaw is that this pattern doesn't communicate that there are errors from code. Just reading an API becomes a game of "guess what's in the box that'll crash your code" - and I don't like that game.
Instead, I prefer errors as values. Combine this with C++'s new [[nodiscard]] attribute, and suddenly we have a way to both communicate there is an error, and we also force people to actually check the error (which is often the biggets complaint about errors as values - no one ever checks them (counterpoint: have you seen try { ... } catch (...) { /* do nothing */ } - no one handles then with exceptions either)). It's clean, the code flow is easy to follow, and there are no surprises.
To support this, I have result<> which can return an error or (optionally) a value. If there's a value returned, the value can be accessed (through C++isms like *res, res->, or standard res.value() and newer res.copy_if_present() and res.move_if_present). If it's an error, then it can be accessed with error(). Casting to a bool will return true on success and false on error, or you can call is_error() to see if it's an error.
Sometimes a value may or may not exist (such as pointers). I added optional<> to codify this stuff, and it has similar access patterns to result<>;
C++ iterators suck. I've written many of them - both readonly and non-readonly versions. They're very verbose, and they offer way more power than is typically needed. If you are trying to make a data structure that performs optimally with the C++ standard library, then yeah, you need C++ iterators. If you're trying to make generic iteration algorithms that are as fast as possible with a wide array of data structures, then yeah you need C++ iterators. If you're trying to write an app that has reasonable performance and avoids unnecessary slowdowns (e.g. O(n^2) linked list iteration by using at()), then C++ iterators are just overkill. This holds true in 90+% of every use case of iterations I actually needed in an application. And in the 10% I needed something better, it was often easier writing the direct access code/extend the data structure/build my own.
To make a simple iterator pattern, I'm borrowing a lot from other languages - in particular Zig. Zig has optionals, and their iterators are simply a function with next() that returns an optional. As for accessing the value, they have a special while construct that copies the value if present - or deletes the value if not. For example:
To do a similar pattern, I made optional and result have copy_if_present and move_if_present which return try if a value was present. This allows us to get something like the following:
If you need to edit elements, then I have pointer iterators as well.
As part of this project, I'm providing additional data structures (or reimplementations of existing data structures but that conform to this library). This section briefly describes them.
A Slice is simply a pointer+len to some data type. Slices have bounds checking and null dereference checking. Slices can be sliced into sub slices. While simple, they form the building block of other data structures and how this whole library operates.
A bitset is a compact list of booleans which also have bitops. Bitsets are defined in terms of uint64_t, so they always have at least 8 bytes of data. For my use cases, this is not excessive memory usage (the header information for a single memory allocation is often heavier than 8 bytes). The advantage of using 64-bit chunks is that bitops and comparisons work on 64-bits at once rather than 8-bits, so there are less ops overall (which is a metric I'm more concerned about).
I also define methods like popcount which counts the total number of bits set, first_set which finds the first set bit (by index), and set_bits which iterates over all the set bits. These methods are useful for keeping track of "dead"/"free" lists in a more compact form than having a bool for every element, and since they operate on 64-bits at a time it's like checking 64-items at once.
A 2d-bitset is basically a 2d grid of on/off values. It's useful for keeping track of black/white pixels, or grid on/off values (e.g. walkable terrain). It's built on top of my bitset and will do a single allocation for the whole 2d array. It doesn't have an array of pointers for the row, instead it does math to convert 2D indexes to a 1D index.
Gen List is a generation-based list. When items in the list are "removed", they're just marked as deleted. When items are added, the list first recycles a deleted item and increases its generation count. Additionally, instead of accessing elements by index, items are accessed by handle. A handle is an id + generation. If the referenced element is dead or has an incorrect generation, then the handle is considered invalid. When items are added, their handle is returned. Additionally, handles for all living elements may be iterated over in addition to iterating over the elements themselves.
A simple first-in-first-out queue. There are two flavors: a fixed queue (fixed capacity, no increase) and a dynamic queue (dynamic capacity, can increase). Both queues are FIFO queues which use a ring buffer under the hood.
A ring buffer that allows pushing in front or back. There are two flavors: fixed (fixed capacity, cannot increase) and dynamic (dynamic capacity, can increase). Has bounds and wraparound checks. Does NOT drop items if full - instead it will return an error result.
A list that's broken up into segments. If the size grows past what the segments can hold, then it will add a new segment. All existing elements remain in place, so this is a memory-stable data structure. Segments double in size with each addition to try to give an O(log(N)) performance. However, this means that there could be a lot of unused memory allocation. To help with this (and preserve the never-move-memory functionality), a method drop_unused_segments is provided which will drop the unused segments at the end of the list. Do note that if there's even a single element in a segment, nothing will happen to that segment. There is no shrinking or reallocating of segments, only dropping what isn't used.
Because of this, it is strongly recommended that you give a good initial capacity estimation to avoid wasting memory.
Several PRNGs are provided as part of the library. These PRNGs all use a VTable to make them interchangeable - which is useful if you need to be swapping them out (e.g. for testing). The following PRNGs are provided:
An IO wrapper with a VTable is provided (VTable allows swapping it out for something else for tests).
There is a default implementation provided which uses the C API. It will try to use 64-bit fopen on Linux.
The traits provided are mostly for ensuring compatibility with the way the library assumes templated types will work (e.g. is an iterator).
The threading library does require the use of threads, and does provide thread safety for its abstractions. The goal of this is to not provide a replacement for std::thread, but to instead provide threading synchronization primitives (e.g. channels).
These allocators are protected by global locks and atomics, so they are thread-safe. Do note that the locking is suboptimal as each type of allocator is protected by a single global lock for that allocator type. At some point I might make it better, but maybe not.
Channels are a communication primitive between threads. They allow one thread to send (or queue) messages and another thread to receive (or dequeu) those messages. Channels have blocking and non-blocking methods, outlined as follows:
Channels can also be closed.
When receiving, you'll get either an optional<T> or a result<optional<T>> (for methods that either timeout or don't block). If the channel is closed, the optional will be a nullopt (no value). Closing a channel does not cause errors on receive.
However, sending to a closed channel will cause there to be an error (hence why all send functions return a result<>).
If there is a timeout, or a non-blocking method would have to block (i.e. cannot lock), then an error will be returned.
Channels have a static size, so they don't need heap allocation.
Inboxes are almost identical to channels, except that they do have a dynamic internal queue which can grow (or shrink).
Pub Sub is a publish/subscribe pattern which uses Senders and Receivers. Internally it does use std::shared_ptr (and the constructor/destructor nonsense required), but externally you don't need to worry about it too much - other than not all the memory will be cleaned up until all subscribers and publishers are removed.
Multiple publishers are supported (as well as multiple subscribers). It doesn't use channels under the hood, instead having a separate mechanism.
This is a mimic of Go's select statement where it will try multiple synchronization operations until one succeeds. For this to work, it only uses non-blocking methods (try_send/try_receive) and iterates over everything in a loop. If nothing matches, it will try to yield the thread to the OS (basically a soft sleep/wait).
My select also supports timeouts as well, and will return an error if it times out. Additionally, it has a "deadlock" detection where if it is only sending to channels and all channels are closed, then it will exit with an error (note that if there).
Traits to make sure any new structs work as expected. Most notable are the following:
1.13.2