MAD-X Roadmap (2011-2013)

The aim of phase 1 was to bring the source code architecture of MAD-X back to a situation "close to normal", allowing better debugging and maintenance, avoiding regression and saving resources. Most of phase-1 tasks were prerequisite to other tasks and required advanced expertise in computer science to deal efficiently with the legacy code. This phase started in October 2011 and took about two years. It was completely transparent to the MAD-X users with no observable behavior change except for some small speed improvement (30-50%) and bugs correction.

The roadmap details describe the list of tasks ordered by priority. Each task is tagged as todo (no tag), ongoing, done and closed corresponding respectively to foreseen, started, achieved (with the approximative end date), and partially archieved (superseded or postponed by another task or project). It is possible that tasks with high priority (appearing first) are not yet started while tasks with low priority (appearing last) are already ongoing. This is because the latter are background tasks that can take benefit of other ongoing tasks running in parallel. Each task has 4-5 sections (Objective, Movitation, Dependency, Appeared, Note), which describe respectively the goal, the task, its dependencies and the first release in which it appears followed by some optional notes.

Split the Monolith (done, 2011.11.21)

Objective:: Increase the cohesion of the code by splitting the monolithic file madxp.c, which contains 40000+ effective lines of code, into smaller files (modules).
Motivation:: A single monolithic file including all other C files is an uncomfortable situation for both compilers and maintainers: (i) compilation time cannot be reduced (always recompile everything); (ii) functions signatures are difficult to cross-check; (iii) functions signatures duplications (no API) lead to inconsistency between caller and callee (stack frame corruption); (iv) the lack of cohesion implies the need of tools like grep, awk or perl to find the information; (v) static analysis tools fail to parse and analyze the code; (vi) build systems are inefficient and brittle on such code architecture.
Dependency:: None. Most other tasks hereafter depend on this reorganization.
Appeared:: 5.00.09. After the split into 66 modules, some tests proved the expected unchanged behavior digit by digit even after this huge restructuration.

Cleanup I/O synchronization (done, 2011.11.09)

Objective:: Replace wrappers and generators by something simpler and more robust.
Motivation:: Python scripts are used to parse all the C and Fortran files and generate wrappers to ensure proper I/O synchronization. The solely role of wrappers is to flush the standard output stream when crossing the border line between C and Fortran worlds. This solution is brittle and does not work on the new files structure. The wrappers and generators should be removed from the project, the code should be cleaned and adapted, and stdout should be configured as unbuffered to solve the problem correctly.
Dependency:: None.
Appeared:: 5.00.09. Results showed perfect synchronization and no performance loss with unbuffered I/O on stdout.

Make a build system (done, 2011.11.28; improved, 2012.06.22)

Objective:: Make a new build system more efficient and robust with the new files structure.
Motivation:: The build system of MAD-X is difficult to modify, brittle to code changes (no dependency management) and not portable (another build system is provided for Windows). Many settings are incompletely supported or confusing and changes in project configuration is error prone. The new system should allow better management of releases, compilers and platform configuration, cross-compilation, parallel compilation, and being multi-platforms. This build system should also allow generic files selections, calculate files dependencies and be easily extensible to perform regression tests in the future.
Dependency:: 'Split the Monolith'. Better after 'Cleanup I/O synchronization' to remove the management of wrappers generators first.
Appeared:: 5.00.09. The new build system fulfill all the requirements and supports 12 compilers (C, C++, F90, F77), 5+ platforms (including native Windows) and 32 & 64 bit architectures. It requires only GNU make and is pretty flexible. This system will be extended in the coming months to perform MAD-X regression tests.

Speed up string manipulation (done, 2011.10.21)

Objective:: Improve the speed of intensively used string manipulation functions.
Motivation:: Many string manipulation functions use algorithms with complexity of O(N^2) instead of O(N). Rewriting these functions should improve the speed of MAD-X since strings are used for (suboptimal) data representation in many places. A much better improvement (robustness & speed) will be to convert all these strings into properly typed objects (phase 2).
Dependency:: None. This task was not prioritary but appeared to be relatively simple after 'Split the Monolith'.
Appeared:: 5.00.09. Results showed a 15-30% speed-up with typical MAD-X scripts.

Make access to program arguments (done, 2011.10.13)

Objective:: Provide access to the program command line arguments.
Motivation:: Command line arguments are convenient to setup programs on a per-run basis without recompiling the code or modifying the scripts. Extending options this way is also simpler than modifying the options supported by the scripting langage. This is of particular importance for improving MAD-X: arguments can setup options for higher debug level, to activate a piece of code for evaluation or to substitute some behavior by another (algorithms, models) before user level integration.
Dependency:: 'Make a build system'. This task was not prioritary but appeared to be relatively simple after 'Split the Monolith'.
Appeared:: 5.00.09.

Cleanup C/Fortan API (done, 2011.11.23)

Objective:: Cleanup the functions signatures shared by C and Fortran.
Motivation:: The functions provided by C and Fortran are spread over many places and their signatures are often unspecified. The observation of stack frame corruption due to inconsistent signatures makes such bug tracking a high-priority. This task is also a good opportunity to exhibit the minimum API provided and required by the two worlds.
Dependency:: 'Split the Monolith' and 'Make a build system' to speed-up compilation time.
Appeared:: 5.00.10.

Cleanup C warnings (done, 2011.12.15)

Objective:: Remove thousands warnings reported by C compilers.
Motivation:: Compilers are one of the primary tools that developers should use for tracking bugs. It is important to correct compilers warnings as they appear since most of them are meaningful. At this stage, MAD-X generates thousands of warning from the C code that should be corrected. The Fortran code should also be process, but lately since even more warnings arose.
Dependency:: 'Split the Monolith' and 'Make a build system' for proper use of compilers.
Appeared:: 5.00.10.

Create MAD repository (done, 2011.12.16)

Objective:: Collect all the materials related to MAD and put it in a single clean place.
Motivation:: The mad account has enough space to hold madx, mad9 and mad8 repositories and websites. As any user account, it has a day-to-day backup easily accessible which allows to repair quickly manipulation errors. On the other hand, the actual repository (/afs/cern.ch/group/si/slap) contains many things collected over the past two decades and not always related to MAD-X. This should be a good opportunity to cleanup the content of the imported directories with some care for HTML and symbolic links.
Dependency:: None.
Appeared:: 5.00.10.

Restore the communication (done, 2011.11.21)

Objective:: Provide tools to enhance the communication within the MAD community.
Motivation:: The communication within the MAD community should be shared and archived for the benefit of the users. The aim is to move from person centric communication (personal emails) to project services (mailing lists, generic email). This should be done by creating e-groups targeting different audience (and bandwidth) and a generic email service mad support.
Dependency:: 'Create MAD repository' for the mad account and email.
Appeared:: 5.00.09.

Make a website (done, 2011.12.08)

Objective:: Provide rapid and easy access to information and materials related to MAD.
Motivation:: The MAD community is lacking a descent website that helps new comers to rapidly access to useful materials and regular users to be informed about evolutions. It should also be the front-end for releases distribution and e-groups access.
Dependency:: 'Create MAD repository' for the mad account and www access.
Appeared:: 5.00.10.

Close modules for functions (done, 2012.01.13)

Objective:: Increase the cohesion and reduce the coupling by exporting only functions part of the module programming interface.
Motivation:: Well designed modules should only export abstract data types and a small set of orthogonal functions (MPI). The monolithic structure of the legacy code has made everything global and shared (worst possible architecture), which favored the use freaky function names to avoid name collisions and spread the implementation of the modules over the entire code (functions must be declared or defined before being used in C). This task will improve the decision taken during the split of the monolith, leading to some extra modules split or merge.
Dependency:: 'Split the Monolith' and 'Make a build system' because this task will require to compile MAD-X about a thousand time.
Appeared:: 5.00.11. This significant task has not affected the name of the exported functions and leave their semantic unchanged. The MPI are often suboptimal and confusing but redesigning them require a major rewrite of MAD-X postponed to phase 2.

Synchronize documentation (done, 2012.01.18)

Objective:: Synchronize the documentation with the MAD-X releases.
Motivation:: The documentation is currently provided in HTML format and it is not part of the versioning system of the MAD-X source code. Hence there is a risk of discrepancy between the documented features and the implemented features with no trace of history. Some attention should be put on the HTML links in the documentation before moving the files to SVN.
Dependency:: 'Make a website' to provide easy access to the table of content and index (shortcuts). 'Create MAD repository' for storage.
Appeared:: 5.00.11. The documentation on the website is updated nightly from SVN.

Setup a trac system (done, 2012.02.01)

Objective:: Provide rapid and easy access to registered defects and requests.
Motivation:: Bug reports and feature requests from users contain information and scripts that must be archived and classified. The archives can be queried for correlated bugs and history and help to find better solutions quicker. The users scripts can be the basis for new regression tests.
Dependency:: 'Split the Monolith' to classify the component that bug belongs to. Better after 'Restore the communication' since bugs reports should be posted on the e-groups or to the generic mad email.
Appeared:: 5.00.12. It was a significant task which allowed to trace back missed bug reports.

Synchronize examples (done, 2012.03.05)

Objective:: Synchronize the examples with the MAD-X releases.
Motivation:: The examples are currently provided in another trunk of the versioning system not synchronized with MAD-X source code. Hence there is a risk of discrepancy between the examples and the releases with no trace of history.
Dependency:: 'Create MAD repository' for storage. Better after 'Provide regression tests' for better complementarity.
Appeared:: 5.00.12. The repository size increases significantly (x10) and needs some file cleaning and sharing. It was the occasion to move all the repository to the new project workspace /afs/cern.ch/project/mad.

PTC as a library (done, 2012.03.14; improved 2012.10.05)

Objective:: Provide PTC as a standalone static library on lxplus.
Motivation:: The PTC source code is used by few projects at CERN, including MAD-X and PTC-ORBIT. The aim is to have only one single source for PTC at CERN and to use the same version in all projects. The MAD-X team proposed to provide a standalone static library (32 bit and 64 bit) for PTC on lxplus in the same way it is done for MAD-X. Other platforms (e.g. Windows, MacOSX) can be supported on demand.
Dependency:: 'Create MAD repository' for storage and 'Make a build system' for building PTC and MAD-X consistently.
Appeared:: 5.00.13. This task demonstrated that the new build system of MAD-X is easily extendable.

Make numdiff (done, 2012.05.30; improved 2012.09.25; extended 2013.04.15)

Objective:: Make a new tool to compare files with numerical content.
Motivation:: The future test system will require to compare files with numerical content. The only portable way to do this in the context of MAD-X is to provide test specific constraints on the potential differences of the results (e.g. cross-platform, cross-compiler). These constraints will be based on the knowledge of the test to ensure that outputs are consistent with references. The comparison will be easy to customize through a configuration file with relative, absolute or input driven (relative) precisions.
Dependency:: 'Make a build system' to build numdiff itself.
Appeared:: 5.00.17.

Make a test system (done, 2012.06.08; improved, 2012.06.22, 2012.09.25)

Objective:: Make a new test system more efficient and robust.
Motivation:: The rapid evolution of the project require hundreds of code compilation per months, which must be tested quickly and accurately to avoid the accumulation of pending bugs. A set of scripts should be run frequently and compared to reference results in order to report defect as quickly as possible. The new test system will be an extension of the new build system and it will require a special tool able to compare numerical results within some customizable precision (relative, absolute or input driven).
Dependency:: 'Split the Monolith', 'Make a build system' and 'Make numdiff'. Better after 'Setup a trac system' to get an overview of bugs history and future requirements.
Appeared:: 5.00.17.

MAD-X as a library (done, 2012.06.22; improved 2012.10.05)

Objective:: Provide MAD-X as a standalone shared library on lxplus.
Motivation:: MAD-X is often used by other projects as a low-level application to build lattices, to compute survey and twiss functions and to perform matching and tracking. Most of these projects use scripting languages like Python to build the necessary glue around MAD-X. The simplest approach is to load libmadx as a shared library and to run MAD-X script using its external interface (mad_init, mad_run, mad_fini) and retrieve the data by direct access to internal variable and tables. Notes that this requires a deep understanding of the MAD-X data structures and internal states (i.e. experts only).
Dependency:: 'Create MAD repository' for storage and 'Make a build system' for building MAD-X and libmadx consistently.
Appeared:: 5.00.17. This task demonstrated once more that the new build system of MAD-X is easily extendable.

Synchronize tests (done, 2012.09.11)

Objective:: Synchronize the tests with the MAD-X releases.
Motivation:: The examples used for testing are currently provided in another trunk of the versioning system not synchronized with MAD-X source code. Hence there is a risk of discrepancy between the tests and the implementation with no trace of history.
Dependency:: 'Create MAD repository' for storage. 'Provide regression tests' for the testsuite.
Appeared:: 5.00.18.

Provide regression tests (done, 2012.09.26; improved 2013.02.20)

Objective:: Provide a set of fast and accurate scripts for regression tests
Motivation:: The new test system must be fed with many regression tests, which should run quickly and test in depth specific features of the implementation. The regression test suite should be run daily and therefore should be efficient, but still accurate. More complicated user tests could be run less often to ensure that large simulations are still consistent with the expected results. This task requires a permanent improvement process.
Dependency:: 'Split the Monolith' and 'Make a test system'. Better after 'Setup a bug tracker' to get an overview of bugs history and future requirements.
Appeared:: 5.00.18.

Close modules for states (closed, 2013.02.20)

Objective:: Increase the cohesion and reduce the coupling by removing global variables.
Motivation:: Well designed modules should not use or export global variables. Global variables are dangerous since they create unexpected side effects. Shared global variables creates an implicit states machine with uncontrolled states transitions leading to many bugs hard to reproduce and very hard to correct since part of the code rely on the side effects. MAD-X use thousands of global variables, most of them being shared. Due to the very high coupling existing in the legacy code, this task will be achievable to some moderate level only (25-50%) and completed only in phase-2.
Dependency:: 'Split the Monolith' and 'Make a build system' because this task will require to compile MAD-X about a thousand time. 'Provide regression tests' to detect unexpected changes in the observable behavior. Better after 'Close modules for functions' since some modules split could be reorganized.
Appeared:: This task is partially completed, and superseded by the new MAD project.

Close modules for types (closed, 2013.02.20)

Objective:: Increase the cohesion and reduce the coupling by only exporting abstract data type.
Motivation:: Well designed modules should only export abstract data types (ADT) and a small set of orthogonal functions (MPI) to work with them. This removes any coupling to the data structure and allows to improve or even substitute the latter by more appropriate data type internally to the module. ADT are essential for module rewrite or substitution. Due to the very high coupling existing in the legacy code, this task will be achievable to some moderate level only (25-50%) and completed in phase-2.
Dependency:: 'Split the Monolith' and 'Make a build system' because this task will require to compile MAD-X about a thousand time. 'Provide regression tests' to detect unexpected changes in the observable behavior. Better after 'Close modules for functions' since some modules split could be reorganized and after 'Close modules for states' since some global variables should disappear.
Appeared:: This task is partially completed, and superseded by the new MAD project.

Setup test environment (closed, 2013.03.13)

Objective:: Automatize builds and tests for all supported platforms and architectures.
Motivation:: The build and test process is a long and tedious process that requires to connect to different servers (one per platforms), setup the appropriate environment and run the build and test system manually. In order to avoid long period without testing the validity of the code, a dedicated test environment should be configured for all the supported platforms (Linux, MacOSX and Windows), 32 and 64 bit architecture and easy access through virtualization. Cross-compilation should allow to build and test 32 bit versions on properly configured 64 bit platforms.
Dependency:: 'Make a build system' and 'Make a test system' to run builds and tests automatically.
Appeared:: 5.01.00, this task is partially completed, and will be superseded by the new build and test infrastructure.

64 bit compliance (closed, 2013.03.13)

Objective:: Makes MAD-X portable on 64 bit architectures.
Motivation:: 64 bit architectures offer more computational power than 32 bit architectures and are nowadays the de facto standard architecture for the supported platforms. Many technical details must solved before one can guarantee that MAD-X is 64 bit compliant, including portability problem also present on 32 bit architectures. This is a background task that may require rewriting significant parts of MAD-X to be completed and hence be postponed to phases 2 and 3.
Dependency:: 'Make a build system' for the many compilations. 'Provide regression tests' to detect any changes in the semantic.
Appeared:: Starting from 5.01.00, MAD-X has a much better compliance with 64 bits.

Setup garbage collector (done, 2013.05.10)

Objective:: Include a garbage collector in MAD to cure memory leaks.
Motivation:: The core of MAD-X is known to leak a lot of memory when interpreting scripts, editing sequence, executing commands or updating element properties. This has been proved to be very penalizing for some studies (e.g. large lattice with versatile parametization) and considered to be impossible to solve without a complete rewrite of the core. The other choosen approach was finally to include a garbage collector to manage the dynamic allocation made by the C code and solve undecidable bugs. The C++ and Fortran code are not supported, but they were not identified as leaking (any?) memory and such support would make the portability of the code much more complex. This major improvement became possible only recently after the test system of MAD-X became mature enough to detect the many instability problems that arose across platforms.
Dependency:: 'Make a test system' to run tests frequently and identify problem arising with the GC.
Appeared:: 5.01.01. This is a major task that would not have been possible without the many test suites.
Note:: The inclusion of a garbage collector in MAD-X is the milestone marking the beginning of a better observable stability of the 64 bit architecture versus the 32 bit architecture (as expected).

Improve the user's guide (ongoing)

Objective:: Reorganize and improve the MAD-X documentation. Check for undocumented and untested features in the source code.
Motivation:: The MAD-X documentation is extremely important for the users. But the current documentation is not optimally structured, the style is minimalistic and very heterogeneous, and the content is incomplete comparing to the implementation of MAD-X. Beside the significant editorial work, it is also important to check for completeness and test uncovered codes and features. This huge task can be combined with the check of code coverity and will only be performed if dedicated resources are assigned to it.
Dependency:: 'Make a website' for easy access to the documentation. 'Check code coverity' for detecting untested and undocumented features.
Appeared:: N/A.

The tasks below are postponed until MAD-X undergoes a significant refactoring

Check code coverity

Objective:: Configure analysis tools to help code quality steering and avoid untested code.
Motivation:: The code of MAD-X has a long history with very heterogeneous programming styles and complexity. Some coding styles are error prone and a source of bugs that are difficult to track and with corrections that require to rewrite significant part of the code (see phase-2). Moreover, many parts of the code have been developed without being systematically tested. Analysis tools help to measure the quality of the code, detect numerous sources of bugs and report code uncovered by tests. This task can only be performed if dedicated resources are assigned to it.
Dependency:: 'Split the Monolith' and 'Set-up test environment' because Coverity needs properly set-up builds and tests.
Appeared:: N/A.

Parallelization compliance

Objective:: Makes MAD-X portable on multi-core processors and multi-cpu architectures.
Motivation:: Multi-core processors and multi-cpu architectures architectures offer much more computational power as soon as the code is parallelizable. Many technical details must solved before one can guarantee that MAD-X is parallelizable. This is a long term task that may require rewriting large parts of MAD-X and hence be postponed to phases 2 and 3.
Dependency:: 'Make a build system' for the many compilations. 'Provide regression tests' to detect any changes in the observable behavior. 'Close modules for states' to enable reentrancy mandatory for parallel execution.
Appeared:: N/A.