Thank you all for your thoughts, they are very insightful, this is exactly the kind of discussion I wanted to have. I think you are all correct, we are just miscommunicating slightly.
Also, sorry for the long mails.


> For the particular issue above, can you play with your -I include paths and add a -D on the command line such that TestMe.cpp is compiled with MockUnwieldy under the hood? Something like
> TestMe.cpp
> mocks/Unwieldy.hpp -> MockUnwieldy.hpp
> gcc -Imocks -DUnwieldy=MockUnwieldy -c TestMe.cpp
> or so?

I have just devised the same solution, altough by doing a #include "TestMe.cpp". Not very elegant, but same idea. I believe both should work in principle. I like yours better. Will play around with this.


> A CLASS IS NOT A UNIT...

Yes, what exactly is a unit? I'm not sure either. A class can be a unit, but not all classes are. There exists at least one hypothetical class that should not be treated as a 'unit' for whatever reason, therefore 'not all classes are units', which may sound equivalent to saying 'a class is not a unit', but these sentences are not equivalent mathematically. But I think we all know what we all mean.


> In general a class is a terrible thing to test in isolation. The purpose of a test is to ensure that no matter what some invariant will never change. Unfortunately we have no way of saying "here is what I mean to test, but the ABI/function signatures may change", and so our test frameworks all end up locking in not just the logic we care about, but also the API and thus prevent changing that code in the future.

Again, depends on the situation. But you are absolutely correct about unintentionally locking into the ABI.


> Thus the smallest "unit" needs to be large enough that whatever refactoring you may want to make in the future (when some future requirement has changed). Now the concern about too large is valid - you need most of your tests to be fast and reliably give the same result 100% of the time. So you do need to find that balance, but be very careful of the temptation to go too small - it will hurt you in the future.

Good point, appreciated. Done that a couple times, and probably will do it some more in the future, accidentally. I try not to.


> One of the largest values of an automated test is when it alerts you to a problem by failing. 80% of your tests (I don't have a study on this, but 80% feels right) will never again fail after the initial failure in TDD and you could safely delete them - but there is no way to know which 20% will fail and so we keep them all. In my experience the large the test the more likely it is to fail on real bugs (they fail on flakiness too), so make sure you have enough large tests - while of course avoiding the issues with large tests - this is often a compromise.

Agreed.


> Any interface with multiple implementations becomes a place to break in tests. You may want to test behavior that is hard to get with the real implementations, but a fake (I avoid mocks - mocks are an assertion that a function will be called - almost never something you care about!) can provide it. I've also had some luck writing code to test the implementations of an interface provide that contract, but not enough to tell you how to do it.

You are right about fakes vs mocks, that is, many times it suffices to write in tests "here's an object, I don't care exactly how it's used, but it should behave like this" to set up the scenario you want to test, which grants freedom for the implementation. But a 'fake' is just a mock, except you go "ALLOW_CALL" instead of "REQUIRE_CALL", or whatever your framework calls it. Of course we can make multiple implementations, but I found it quite handy to use mocks as fakes. The focus is on controlled behavior, regardless of application. This is what I meant by 'mocking'. Sorry, this wasn't clear.


> Just remember you still need extensive manual release testing. Automated tests long term save more money than they cost, and they speed up release, but they only get a minority of the issues manual testing will catch.

Agreed, but this too varies from software to software. Currently I am working on a software which we cannot really do 'manual testing' on, it is literally impossible, but I suppose this is an exception rather than the norm. It's hard to make a point for something (more 'unit' tests) without implicitly deemphasizing the alternatives. This was not my intent.


> There is good reason for that. Most of us do not need anywhere near as generic as library code and so we don't need that, while the library does. Most of us don't write code where a 0.1% improvement would have a noticeable effect, but for library code that small a change will make a big difference to somebody, and even you personally don't see the difference it will save millions of dollars every year in the electric bills (across many different companies - Facebook alone has hinted that they can save 6 figures from changes of that magnitude) Most of us don't write the complex algorithms - we look for them in our library.

This is true without doubt. I did not mean to suggest these features aren't useful or important, and I do appreciate them. I just wish there were also features that target the 'average C++ programmer', and would allow us to write better code more easily (express intent, the 'what' instead of the 'how'), because at the end of the day, someone has to be there to use those awesome libraries. There's new code to be written, and existing code to be maintained, and I personally believe that a new language (or library, doesn't really matter) feature for this would be really useful, and would fill a real gap. Obviously if the consensus is that there are much bigger issues to be resolved in the C++ standard right now, I understand that.


I also said previously that, tests that are done with unit testing frameworks, regardless of the label we assign to them ('unit' or not, let's call them UTest for simplicity) are 'far preferred' to other, higher levels of testing. What I meant is that, if it was possible to 'cover the same ground' with UTests, as in, higher level automated tests would bring no additional benefits at all, then the UTests are preferred due to their (supposed) simplicity and reliability. Here's a concrete example. We are testing a heavily multithreaded software. Let's say that request X is handled by sending 3 sub-requests to other components, wait for their results, do something with them, and then send the reply that X was fulfilled. Here's an invariant: "Regardless of the timing and order in which the 3 sub-requests are completed, the end result should be the same". With system-level automated tests, we have little control over that, it depends on the OS kernel (thread scheduling), other external factors and God's will probably, so to us it's random. In a UTest, we might be able to reliably reproduce race conditions, unfortunate timings, etc, and so UTests win IF we do not consider anything else. Obviously in this case, they are not equivalent, the system-level test does cover a lot more than just this, but the issue is, if we cannot easily create the UTest (too much refactor, too much mocking and boilerplate, not enough resources, not worth it), we will skip it, and take solace in the fact that at least we have the automated tests. And I sincerely believe that on this project (my daily full-time job) about 80% of all bugs, crashes, sporadic behaviors, whatever, fall into this category. And the issue is, even if we find the cause of the bug, it still takes too much effort to refactor the code for a UTest, so we just don't test it.


> You can write reusable types with programming techniques such as dependency injection just fine; structural types don't seem necessary for this.

I believe techniques such as dependency injection, should be used only if the underlying coding problem requires it. If I am forced to employ such techniques to make the code testable, then to me this is a sign of a missing language-level feature (or a missing library, whatever). I shouldn't need to go out of my way and litter the production code with pure virtual interfaces and 'fake polymorphism' (only one real implementation, all others are mocks/fakes for testing) or std::functions, or your choice of dependency injection/type erasure, etc, just because I want to add a missing test case. It is also valid to say that the 'underlying coding problem' actually involves considering testability, and that there should have been pure virtual interfaces all along, we were just too lazy to make them. I don't agree with this though.


Thinking about this problem a bit more, I feel like the issue is with high-level (high abstraction) C++ code. Yes, it is the beauty and selling point of C++ that we can get down to individual bytes while supporting many paradigms and levels of abstraction. It's a real swiss army knife. But there comes a point beyond which I don't really care for that, I don't want to deal with base class pointers and types, I just want my damn class to call a method named X on another class, and I want to test it. I do realize that I cannot have the cake and eat it too. Nominal typing is a double edged sword. I just feel like there's plenty of C++ code out there that fall into this category, and the language is not trying to accommodate for this. A language like Python is the exact opposite, and I would much rather implement such high-level control logic in Python than in C++. It's interesting though that by thinking about a possible, minimal viable solution to this issue, I re-discovered the notion and benefits of structural typing.


> Structural typing seems like a really bad fit for C++. It makes more sense in a language like TypeScript where all members of a type are accessed by name, and their order within the structure is irrelevant for parameter types. In C++, a struct containing a bool and int could not be structurally equivalent to a struct containing an int and a bool.
> This seems like something that would add a lot of complexity while not getting anywhere near the benefits you see in other languages. Not to mention, C++ is already an absurdly complex language as is, and the features we now add have to give us really good returns for the complexity investment.

So far, I feel like the general consensus is "this is the C++ way of doing things, just deal with it", to which I say I think we could do better. C++20 concepts were a step towards structural typing support. I agree that full-on structural typing would be a bad fit for C++, but perhaps a teeny-tiny feature, like my proposal, might just be worth it. I don't know. I am starting to change my mind in favor of the double compilation solution at the beginning of this mail though.