The ctests are not convenient tests. In this file we can attempt to log the benefits and costs to using them so we can periodically evaluate which tests we should keep and which we should get rid of. Below are some objective costs: - The time it takes to run the ctests. - The email discussion back-and-forth. There are also some non-objective costs. The ctests are annoying and not fun to maintain, discuss, and fix. Although it's hard to measure this "annoyance" cost, it should be taken into account when we review. Below we can try to keep a log of which bugs are found with which ctests. This way we can see which tests are the most useful. It is difficult to objectively differentiate between minor bugs and important bugs. The "Benefit" row gives the amount of days in-between when an issue is found because of the ctests and when an issue is reported by someone not using the ctests. The "ctest" row gives a ctest pattern for the tests that were affected by the regression. Examples where convergence tests took time and annoyance to figure out that they were actually "false positives" (i.e., no bug that is worth the time or complexity to fix): Report : https://www.mail-archive.com/search?l=mid&q=20210125200655.e27xaaoy2belyxhw%40tallinn Report : https://www.mail-archive.com/lyx-devel@lists.lyx.org/msg218248.html Testing compilation after lyx2lyx roundtrip was not helpful in this case: report : https://www.mail-archive.com/lyx-devel@lists.lyx.org/msg218232.html ctests : de/UserGuide_lyx(22|23) Examples where an upstream issue was reported and ended up wasting time: Report : https://github.com/latex3/latex2e/issues/836#issuecomment-1159826566 ctests : export/templates/Obsolete/.*American_Geophysical_Union.*_(lyx22|lyx23|pdf|dvi).* Examples where bugs were fixed (more quickly) because of ctests: Report : https://www.mail-archive.com/search?l=mid&q=20210126190837.pwuxhrhrikzszfrh%40tallinn Fix : 92f8612a Report : https://www.mail-archive.com/search?l=mid&q=b0a33192bc5635bd1cb5596e346f9c34615ffeb4.camel%40lyx.org Fix : ab1d418f Fix : ef2b2827 Fix : 1ddc1c6b Report : https://www.mail-archive.com/search?l=mid&q=20210218051838.jbwnemfv5ilhlsls%40tallinn Fix : 358e4ace Fix : 3df42924 Fix : 67d0b80f Fix : 8ce6c815 Report : https://www.mail-archive.com/search?l=mid&q=20210523014926.xewdr5x62br2okvy%40tallinn Fix : 39a9e370 Report : https://www.mail-archive.com/search?l=mid&q=20210528001413.56phju7dvfn4dexs%40tallinn Fix : ffa6a724 Report : https://www.mail-archive.com/search?l=mid&q=20210529174524.idnmt5w4tvhuvyy6%40tallinn Note1 : We are still not sure what to do regarding how to make it more user-friendly for users to realize that "-dALLOWPSTRANSPARENCY" is needed for the ps2pdf converter. Benefit: 14 days (https://www.lyx.org/trac/ticket/12303) Report : https://www.mail-archive.com/search?l=mid&q=20211205190238.edrtqmg3fkdr7gbo%40poole Fix : 489bf46a, e3489d54 Report : https://www.mail-archive.com/search?l=mid&q=20220205184128.4clckanaufwmx5z4%40gary Fix : d66c27de Report : https://www.mail-archive.com/search?l=mid&q=20220206142559.rpgwki7wruhtasxp%40gary Fix : f71f71d0 Report : https://www.mail-archive.com/search?l=mid&q=20220310131706.qdthdjblfpme2wew%40gary Fix : dc502c82 ctests : examples/Welcome_pdf2 report : https://www.mail-archive.com/lyx-devel@lists.lyx.org/msg218232.html Fix : b8502a3e, e316e21a ctests : de/UserGuide_lyx(16|20|21) report : https://www.mail-archive.com/search?l=mid&q=20221218210348.gkwloeu74zjewqsu%40gary Fix : c2a324e24 ctests : DEFAULTOUTPUT_export/export/latex/CJK/zh_CN-en-de-el-ru_language-default_pdf2 Examples where upstream (e.g., LaTeX packages) bugs were fixed (more quickly) because of ctests: Fix : https://github.com/nrnrnr/noweb/pull/12#event-4892099481 Report : (private message to Jürgen: Message-ID:<20211205192054.zbxij5butj3wdyvf@poole>) Fix : some of the polyglossia commits in the range 1c36dcdc^...bf69288c Benefit: Not clear. The regressions were in the development version so they could have easily been caught before release. Report : https://www.mail-archive.com/search?l=mid&q=20211207200100.o27ojaftehrjfcza%40poole Fix : 2eaf30c5 Report : (private message to Jürgen: LyXMessage-ID:<20220216020703.la2fd5gu7yz7y7dc@gary>) Fix : https://github.com/reutenauer/polyglossia/commit/c7489ebaf8c66a7e4fe2529de7068ebb79c68812 Benefit: Not clear. The regressions were in the development version so they could have easily been caught before release. ctests : export/export/lyx2lyx/revert-languages-polyglossia-primary_lyx22 Report : https://github.com/latex3/latex2e/issues/836 Fix : https://github.com/latex3/latex2e/commit/878a4715df024fc3425d2bfd5d3d138b9395c178 ctests : doc/EmbeddedObjects Report : https://github.com/gsilano/EuropeCV/issues/29 Fix : none yet ctests : Modern_CV Report : https://github.com/schlcht/microtype/issues/21 upstream: https://github.com/latex3/latex2e/issues/880 Fix : none yet ctests : export/templates/Articles/Hebrew_Article_.*(dvi|pdf|pdf2|pdf3) Report : Message-ID: Report : Message-ID: ctests : (export/export/WrongDfnTagHandling_lyx23|export/examples/Welcome_lyx23)