Date: Mon, 1 Sep 2025 19:00:52 -0700
On Mon, Sep 1, 2025 at 4:50 PM Henry Miller via Std-Proposals
<std-proposals_at_[hidden]> wrote:
>
>
> I debated sending this to the list, but since my reply is off topic and potentially could be seen as an attack (though I don't mean it that way I figured I'm better send this privately. Feel free to forward to others - I think you will find it helps you avoid jumping into a world of hurt.
>
> On Sat, Aug 30, 2025, at 00:40, Tek, Robert Mate via Std-Proposals wrote:
>
> >
> > Mind you, when I say 'unit test', I am not necessarily referring to
> > 'proper' unit testing of a single class in isolation, but rather any
>
> A CLASS IS NOT A UNIT. I don't know what unit means - I've pressed people on this and never got a satisfactory definition (and thus I quit using the term unit or integration tests as they are too meaningless), but everyone seems to agree a class is not a unit. A class can be multiple units without violation the single responsibility principal.. More often a unit is a collection of classes and sometimes even modules, and yet still not an integration test (again, whatever that is).
A class has invariants, and those should be tested in that exercising
the public interface of the class that the invariant holds before and
after manipulating the public interface. In that sense, a class is a
unit. It may contain other objects which themselves may be
unit-tested.
> In general a class is a terrible thing to test in isolation. The purpose of a test is to ensure that no matter what some invariant will never change. Unfortunately we have no way of saying "here is what I mean to test, but the ABI/function signatures may change", and so our test frameworks all end up locking in not just the logic we care about, but also the API and thus prevent changing that code in the future.
>
The API and ABI are contracts with the caller. That caller may be an
entirely different organization and thus arbitrary changes to either
the API or ABI should be noted in the tests. Not necessarily to
prevent them from changing, but to at least be aware when they are.
And thus not inadvertently shipping/publishing the new release with a
changed API or ABI.
> Thus the smallest "unit" needs to be large enough that whatever refactoring you may want to make in the future (when some future requirement has changed). Now the concern about too large is valid - you need most of your tests to be fast and reliably give the same result 100% of the time. So you do need to find that balance, but be very careful of the temptation to go too small - it will hurt you in the future.
>
> One of the largest values of an automated test is when it alerts you to a problem by failing. 80% of your tests (I don't have a study on this, but 80% feels right) will never again fail after the initial failure in TDD and you could safely delete them - but there is no way to know which 20% will fail and so we keep them all. In my experience the large the test the more likely it is to fail on real bugs (they fail on flakiness too), so make sure you have enough large tests - while of course avoiding the issues with large tests - this is often a compromise.
I disagree. The passing tests are exercising things so that
inadvertent changes don't break things. If one deletes these passing
tests after a release or two, then regressions may creep into the
code.
> We have learned some rules over the years.
>
> UI is partially fashion/fads and so it changes often, so you should always test just below the UI. There are some great UI testing tools which you should use to do "integration tests" as well if you have such a tool, but in general test below the UI. This is also a good thing because if you have a strong UI layer division you can add other UIs and this is often useful.
I'm not a UI guy, so I don't feel the pain of attempting to test a UI
(beyond simple terminal CLIs).
> Anything that has '"many" users won't be changed anyway, even if they are internal. While you could in theory make a change in practice anytime you look at it you discover all those places that need to be touched and give up (even with automatic refactoring you still need to manually test it all - I believe in automated tests, but good manual testing is still very important if you want high quality). As such you may as well test that extensively as it won't change anyway. I would test "strlen" as a unit because despite how bad c strings are, they are not going anywhere, but if I was implementing new a string interface for my company I'd test it mostly through the code that uses it until I have a lot of users at which point the interface can't/won't change anymore.
Or work in a company that has invested in a good testing
infrastructure, and a culture of maintaining those tests. Yes, they
exist.
> Any interface with multiple implementations becomes a place to break in tests. You may want to test behavior that is hard to get with the real implementations, but a fake (I avoid mocks - mocks are an assertion that a function will be called - almost never something you care about!) can provide it. I've also had some luck writing code to test the implementations of an interface provide that contract, but not enough to tell you how to do it.
Depends on what you consider a "mock" I suppose. I have seen some
"mocking" frameworks that exist only to count function calls. I tend
to think that any extra implementations of an interface are mocks of
the real interface. Perhaps instead of consulting a database, it just
reads from a static array inside the mock implementation. That sort
of thing.
> Next I'd look at the org chart. There is always friction around code that is used by other teams, and changing those interfaces becomes political. Thus I would always test my code to others before handing it off, changes will become hard anyway (even if we are designing the interface together it will be locked in ways that I'd refactor internals code). The farther the team is from me on the org chart the more testing I will do.
>
> > kind of testing that can be done with unit testing frameworks such as
> > Google Test or Catch2, because we all know that many times, these tests
> > are more on the level of integration tests, or sit somewhere above
> > strict unit testing level on the testing pyramid. Regardless, any such
> > tests are welcome, and better than not testing at all, and (in my
> > experience) are far preferred than release testing (testing the entire
> > codebase all at once).
>
> Just remember you still need extensive manual release testing. Automated tests long term save more money than they cost, and they speed up release, but they only get a minority of the issues manual testing will catch.
Or a culture of automated tests. And mean it. Where I work, there's
too many different devices to even consider trying to do manual
testing. There are some parts that are manually tested, but those are
way beyond even the level of even integration testing and are more
customer-scale testing.
I guess the TL;DR is: I don't agree with the emphasis you appear to be
putting on manual testing, and the de-emphasizing of automated
testing.
<std-proposals_at_[hidden]> wrote:
>
>
> I debated sending this to the list, but since my reply is off topic and potentially could be seen as an attack (though I don't mean it that way I figured I'm better send this privately. Feel free to forward to others - I think you will find it helps you avoid jumping into a world of hurt.
>
> On Sat, Aug 30, 2025, at 00:40, Tek, Robert Mate via Std-Proposals wrote:
>
> >
> > Mind you, when I say 'unit test', I am not necessarily referring to
> > 'proper' unit testing of a single class in isolation, but rather any
>
> A CLASS IS NOT A UNIT. I don't know what unit means - I've pressed people on this and never got a satisfactory definition (and thus I quit using the term unit or integration tests as they are too meaningless), but everyone seems to agree a class is not a unit. A class can be multiple units without violation the single responsibility principal.. More often a unit is a collection of classes and sometimes even modules, and yet still not an integration test (again, whatever that is).
A class has invariants, and those should be tested in that exercising
the public interface of the class that the invariant holds before and
after manipulating the public interface. In that sense, a class is a
unit. It may contain other objects which themselves may be
unit-tested.
> In general a class is a terrible thing to test in isolation. The purpose of a test is to ensure that no matter what some invariant will never change. Unfortunately we have no way of saying "here is what I mean to test, but the ABI/function signatures may change", and so our test frameworks all end up locking in not just the logic we care about, but also the API and thus prevent changing that code in the future.
>
The API and ABI are contracts with the caller. That caller may be an
entirely different organization and thus arbitrary changes to either
the API or ABI should be noted in the tests. Not necessarily to
prevent them from changing, but to at least be aware when they are.
And thus not inadvertently shipping/publishing the new release with a
changed API or ABI.
> Thus the smallest "unit" needs to be large enough that whatever refactoring you may want to make in the future (when some future requirement has changed). Now the concern about too large is valid - you need most of your tests to be fast and reliably give the same result 100% of the time. So you do need to find that balance, but be very careful of the temptation to go too small - it will hurt you in the future.
>
> One of the largest values of an automated test is when it alerts you to a problem by failing. 80% of your tests (I don't have a study on this, but 80% feels right) will never again fail after the initial failure in TDD and you could safely delete them - but there is no way to know which 20% will fail and so we keep them all. In my experience the large the test the more likely it is to fail on real bugs (they fail on flakiness too), so make sure you have enough large tests - while of course avoiding the issues with large tests - this is often a compromise.
I disagree. The passing tests are exercising things so that
inadvertent changes don't break things. If one deletes these passing
tests after a release or two, then regressions may creep into the
code.
> We have learned some rules over the years.
>
> UI is partially fashion/fads and so it changes often, so you should always test just below the UI. There are some great UI testing tools which you should use to do "integration tests" as well if you have such a tool, but in general test below the UI. This is also a good thing because if you have a strong UI layer division you can add other UIs and this is often useful.
I'm not a UI guy, so I don't feel the pain of attempting to test a UI
(beyond simple terminal CLIs).
> Anything that has '"many" users won't be changed anyway, even if they are internal. While you could in theory make a change in practice anytime you look at it you discover all those places that need to be touched and give up (even with automatic refactoring you still need to manually test it all - I believe in automated tests, but good manual testing is still very important if you want high quality). As such you may as well test that extensively as it won't change anyway. I would test "strlen" as a unit because despite how bad c strings are, they are not going anywhere, but if I was implementing new a string interface for my company I'd test it mostly through the code that uses it until I have a lot of users at which point the interface can't/won't change anymore.
Or work in a company that has invested in a good testing
infrastructure, and a culture of maintaining those tests. Yes, they
exist.
> Any interface with multiple implementations becomes a place to break in tests. You may want to test behavior that is hard to get with the real implementations, but a fake (I avoid mocks - mocks are an assertion that a function will be called - almost never something you care about!) can provide it. I've also had some luck writing code to test the implementations of an interface provide that contract, but not enough to tell you how to do it.
Depends on what you consider a "mock" I suppose. I have seen some
"mocking" frameworks that exist only to count function calls. I tend
to think that any extra implementations of an interface are mocks of
the real interface. Perhaps instead of consulting a database, it just
reads from a static array inside the mock implementation. That sort
of thing.
> Next I'd look at the org chart. There is always friction around code that is used by other teams, and changing those interfaces becomes political. Thus I would always test my code to others before handing it off, changes will become hard anyway (even if we are designing the interface together it will be locked in ways that I'd refactor internals code). The farther the team is from me on the org chart the more testing I will do.
>
> > kind of testing that can be done with unit testing frameworks such as
> > Google Test or Catch2, because we all know that many times, these tests
> > are more on the level of integration tests, or sit somewhere above
> > strict unit testing level on the testing pyramid. Regardless, any such
> > tests are welcome, and better than not testing at all, and (in my
> > experience) are far preferred than release testing (testing the entire
> > codebase all at once).
>
> Just remember you still need extensive manual release testing. Automated tests long term save more money than they cost, and they speed up release, but they only get a minority of the issues manual testing will catch.
Or a culture of automated tests. And mean it. Where I work, there's
too many different devices to even consider trying to do manual
testing. There are some parts that are manually tested, but those are
way beyond even the level of even integration testing and are more
customer-scale testing.
I guess the TL;DR is: I don't agree with the emphasis you appear to be
putting on manual testing, and the de-emphasizing of automated
testing.
Received on 2025-09-02 02:01:10