2021-01-26 18:19:34 +00:00
|
|
|
The ctests are not convenient tests. In this file we can attempt to log the
|
|
|
|
|
benefits and costs to using them so we can periodically evaluate which tests we
|
|
|
|
|
should keep and which we should get rid of.
|
|
|
|
|
|
|
|
|
|
Below are some objective costs:
|
|
|
|
|
|
|
|
|
|
- The time it takes to run the ctests.
|
|
|
|
|
- The email discussion back-and-forth.
|
|
|
|
|
|
|
|
|
|
There are also some non-objective costs. The ctests are annoying and not fun to
|
|
|
|
|
maintain, discuss, and fix. Although it's hard to measure this "annoyance"
|
|
|
|
|
cost, it should be taken into account when we review.
|
|
|
|
|
|
|
|
|
|
Below we can try to keep a log of which bugs are found with which ctests. This
|
|
|
|
|
way we can see which tests are the most useful. It is difficult to objectively
|
|
|
|
|
differentiate between minor bugs and important bugs.
|
2021-01-26 18:28:41 +00:00
|
|
|
|
2021-01-28 05:14:37 +00:00
|
|
|
Examples where convergence tests took time and annoyance to figure out that
|
|
|
|
|
they were actually "false positives" (i.e., no bug that is worth the time or
|
|
|
|
|
complexity to fix):
|
2021-01-26 18:28:41 +00:00
|
|
|
|
|
|
|
|
https://www.mail-archive.com/search?l=mid&q=20210125200655.e27xaaoy2belyxhw%40tallinn
|
2021-01-28 05:25:30 +00:00
|
|
|
|
|
|
|
|
|
2021-03-07 01:09:32 +00:00
|
|
|
Examples where bugs were fixed (more quickly) because of ctests:
|
2021-01-28 05:25:30 +00:00
|
|
|
|
|
|
|
|
Report : https://www.mail-archive.com/search?l=mid&q=20210126190837.pwuxhrhrikzszfrh%40tallinn
|
|
|
|
|
Fix : 92f8612a
|
2021-01-28 16:59:41 +00:00
|
|
|
|
|
|
|
|
Report : https://www.mail-archive.com/search?l=mid&q=b0a33192bc5635bd1cb5596e346f9c34615ffeb4.camel%40lyx.org
|
|
|
|
|
Fix : ab1d418f
|
2021-02-06 03:45:14 +00:00
|
|
|
|
|
|
|
|
Fix : ef2b2827
|
2021-02-08 05:13:45 +00:00
|
|
|
|
|
|
|
|
Fix : 1ddc1c6b
|
2021-02-20 00:19:12 +00:00
|
|
|
|
|
|
|
|
Report : https://www.mail-archive.com/search?l=mid&q=20210218051838.jbwnemfv5ilhlsls%40tallinn
|
|
|
|
|
Fix : 358e4ace
|
2021-03-07 01:09:24 +00:00
|
|
|
|
|
|
|
|
Fix : 3df42924
|
2021-04-10 16:17:16 +00:00
|
|
|
|
|
|
|
|
Fix : 67d0b80f
|
2021-04-24 03:55:13 +00:00
|
|
|
|
|
|
|
|
Fix : 8ce6c815
|
2021-05-26 05:20:45 +00:00
|
|
|
|
|
|
|
|
Report : https://www.mail-archive.com/search?l=mid&q=20210523014926.xewdr5x62br2okvy%40tallinn
|
|
|
|
|
Fix : 39a9e370
|
2021-05-28 14:35:26 +00:00
|
|
|
|
|
|
|
|
Report : https://www.mail-archive.com/search?l=mid&q=20210528001413.56phju7dvfn4dexs%40tallinn
|
|
|
|
|
Fix : ffa6a724
|
2021-06-20 04:05:42 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
Examples where upstream (e.g., LaTeX packages) bugs were fixed (more quickly)
|
|
|
|
|
because of ctests:
|
|
|
|
|
|
|
|
|
|
Fix : https://github.com/nrnrnr/noweb/pull/12#event-4892099481
|
