lyx_mirror/development/autotests/ctests-costs-benefits.txt
2022-06-10 09:16:55 -04:00

111 lines
4.0 KiB
Plaintext

The ctests are not convenient tests. In this file we can attempt to log the
benefits and costs to using them so we can periodically evaluate which tests we
should keep and which we should get rid of.
Below are some objective costs:
- The time it takes to run the ctests.
- The email discussion back-and-forth.
There are also some non-objective costs. The ctests are annoying and not fun to
maintain, discuss, and fix. Although it's hard to measure this "annoyance"
cost, it should be taken into account when we review.
Below we can try to keep a log of which bugs are found with which ctests. This
way we can see which tests are the most useful. It is difficult to objectively
differentiate between minor bugs and important bugs.
The "Benefit" row gives the amount of days in-between when an issue is found
because of the ctests and when an issue is reported by someone not using the
ctests.
The "ctest" row gives a ctest pattern for the tests that were affected by the
regression.
Examples where convergence tests took time and annoyance to figure out that
they were actually "false positives" (i.e., no bug that is worth the time or
complexity to fix):
https://www.mail-archive.com/search?l=mid&q=20210125200655.e27xaaoy2belyxhw%40tallinn
Examples where bugs were fixed (more quickly) because of ctests:
Report : https://www.mail-archive.com/search?l=mid&q=20210126190837.pwuxhrhrikzszfrh%40tallinn
Fix : 92f8612a
Report : https://www.mail-archive.com/search?l=mid&q=b0a33192bc5635bd1cb5596e346f9c34615ffeb4.camel%40lyx.org
Fix : ab1d418f
Fix : ef2b2827
Fix : 1ddc1c6b
Report : https://www.mail-archive.com/search?l=mid&q=20210218051838.jbwnemfv5ilhlsls%40tallinn
Fix : 358e4ace
Fix : 3df42924
Fix : 67d0b80f
Fix : 8ce6c815
Report : https://www.mail-archive.com/search?l=mid&q=20210523014926.xewdr5x62br2okvy%40tallinn
Fix : 39a9e370
Report : https://www.mail-archive.com/search?l=mid&q=20210528001413.56phju7dvfn4dexs%40tallinn
Fix : ffa6a724
Report : https://www.mail-archive.com/search?l=mid&q=20210529174524.idnmt5w4tvhuvyy6%40tallinn
Note1 : We are still not sure what to do regarding how to make it more user-friendly
for users to realize that "-dALLOWPSTRANSPARENCY" is needed for the ps2pdf converter.
Benefit: 14 days (https://www.lyx.org/trac/ticket/12303)
Report : https://www.mail-archive.com/search?l=mid&q=20211205190238.edrtqmg3fkdr7gbo%40poole
Fix : 489bf46a, e3489d54
Report : https://www.mail-archive.com/search?l=mid&q=20220205184128.4clckanaufwmx5z4%40gary
Fix : d66c27de
Report : https://www.mail-archive.com/search?l=mid&q=20220206142559.rpgwki7wruhtasxp%40gary
Fix : f71f71d0
Report : https://www.mail-archive.com/search?l=mid&q=20220310131706.qdthdjblfpme2wew%40gary
Fix : dc502c82
ctests : examples/Welcome_pdf2
Examples where upstream (e.g., LaTeX packages) bugs were fixed (more quickly)
because of ctests:
Fix : https://github.com/nrnrnr/noweb/pull/12#event-4892099481
Report : (private message to Jürgen: Message-ID:<20211205192054.zbxij5butj3wdyvf@poole>)
Fix : some of the polyglossia commits in the range 1c36dcdc^...bf69288c
Benefit: Not clear. The regressions were in the development version so they
could have easily been caught before release.
Report : https://www.mail-archive.com/search?l=mid&q=20211207200100.o27ojaftehrjfcza%40poole
Fix : 2eaf30c5
Report : (private message to Jürgen: LyXMessage-ID:<20220216020703.la2fd5gu7yz7y7dc@gary>)
Fix : https://github.com/reutenauer/polyglossia/commit/c7489ebaf8c66a7e4fe2529de7068ebb79c68812
Benefit: Not clear. The regressions were in the development version so they
could have easily been caught before release.
ctests : export/export/lyx2lyx/revert-languages-polyglossia-primary_lyx22
Report : https://github.com/latex3/latex2e/issues/836
Fix : https://github.com/latex3/latex2e/commit/878a4715df024fc3425d2bfd5d3d138b9395c178
ctests : doc/EmbeddedObjects
Report : https://github.com/gsilano/EuropeCV/issues/29
Fix : none yet
ctests : Modern_CV