ROOT retreat 24.9.2015
Philippe, Bertrand, Axel, Enric, Danilo, Olivier, Gerri, Lorenzo, David, Pere, Benedikt, Liza, Enrico, John (up to "Collaboration" discussion)
Discussion items taken from here
- Decide the retirement root-talk mailing list. Keep it only for announcements.
- agreement to drop root-talk for discussions once there is a forum set up that supports a mail interface
- Decide the replacement of the current Forum with a new one having a mailer interface. Can 'Discourse Forum' do the work?
- Axel showing "discourse" as a candidate. Will set up a prototype service until end of the year. ( http://www.discourse.org )
- Decide on the continuation of the ‘user support’ weekly shifts.
- agreed to keep the shifts, but with more advanced planning of shift schedule
- discussion on role of shifter;
- all team members should make sure that there are proper response times to questions
- Investigate ways to promote more the FAQs and facilitate their authoring.
- move FAQ into forum as to have a unique place to look for it. To be done in the new discourse based forum.
Collaboration and Contributors
- Objective to make the contributions as easy as possible. With clear instructions and process including code reviews.
- **see https://root.cern.ch/collaborate-with-us **
- add a list of low-hanging-fruits (could flag items in JIRA as open for others).
- more instructions on full process including how to write tests, execute them, etc
- make the collaborate-with-us page more visible, e.g. as first level item in the menu
- Clarify what will be the responsibilities the core development team keeps and what we would like to open to contributors. In particular, and maybe to be discussed about CERN (SFT? SFT+IT-DSS+IT-SDC(?)?) members only, what are the core responsibilities we want to keep at CERN and/or in the group and the ones we want to collaborate with the external world.
- agreement that responsibility for core stays with current ROOT team; contributions obviously welcome
- need to make sure that proper recognition to outside contributors is given
- Decide what is our strategy for assigning new tasks and responsibilities to SFT,IT-*, US-CMS/FNAL, Diana or HSF members.
- see above
- Decide if we want to seriously invest in the support of non HEP-science and, if yes, how we can concile this direction with the objectives of CERN.
- agreement to better integrate in non HEP-science
- should spend 10% on outward looking activities according according to John
- Decide If "ROOT as a Service" becomes a new line of work, and since this implies a collaboration with IT, we need to clarify our role and IT's role in such collaboration.
- agreement to provide a 3 page 'proposal document' to IT and collaborate for the development of a pilot service/prototype. Limit work on development
- long-term future to be decided after user feedback on pilot service
Modularization and BOOT
- Agree that the modularization as presented several occasions is a re-prerequisite for attracting collaboration
- consensus to go into direction of modularization as it clarifies responsibilities and allows users to select what they want
- requirements to be made very clear
- danger is to have competing installation mechanisms for components (ROOT internal vs. distribution)
- Decide what are the requirements for BOOT and attach a timescale to the project as well as identify a rough small set of intermediate deliverables assuming the effort we can afford.
- Pere will work on that; results in 6 months from now.
- Encourage contributors that are packaging ROOT in standard distributions (ubuntu, Mac, fedora, etc.). Collect information and provide recognition.
- agreement to have a visible list of contributors that do the packaging of ROOT. Helps understand what is done. Gives recognition to people.
Interoperability with other languages and libraries
- Decide what to do with PyROOT now that the contribution of Wim is to be considered at the "best effort" level
- need to find a new contributor urgently
- could look for volunteers; propose migration to Python 3 as catching project
- Decide about the interoperability of ROOT with other libraries and languages. We could do that discussing at different levels:
- encourage contributions from outside; no active commitment by the team
- Python packages: we could decide if we want to investigate the integration in the math area (scikit, numpy, sympy), in the IO area (pandas), in the UI area (ROOTPy), in the graphics area (Matplotlib).
- core functionality should be provided by ROOT itself and not be replaced; core functionality are things such as I/O, histograms, graphics
- IPython: we could decide if the ipython shell shall become a 1st class citizen, if the notebooks shall become our main documentation format and a co-primary interactive analysis interface
- should work out of the box without intervention by the ROOT team
- We could decide or "decide to study in order to be able to decide later" if R deserves a level of interoperability at the level of Python
- need to have a closer look at why R is much more successful than ROOT. Then decide.
- Regarding IPython, we also need to discuss how to integrate it with ROOT (e.g. builtin module) in order to facilitate as much as possible the local installation and tests with ROOT notebooks. This also includes the discussion about how to instantiate the local server (i.e. root notebook command).
- leave installation up to users; give reasonable help message if missing
- Decide the level of interoperability with Julia as a language for scientific software.
- provide bindings and wait for user contributions
- Decide on the completion of the Doxygen. What needs still to be done.
- 1/3 of things done. Migration will be completed
- Decide if we want to continue maintaining the users' guide or if we think a different approach to documentation could be beneficial. If we decide to keep it, we could decide how to tackle its rewriting since, as of now, it is in a less than optimal shape.
- Clarification of the role and presentation of the ‘how to’, ‘tutorials’, ‘gallery codelets’, ‘notebooks’, ‘FAQs’,
- auto-generated gallery with examples for every single display option
- gallery contains source code for how to create the plots
- take advantage of notebooks wherever possible
- integrate the material into nightly tests
- Decide if we write a paper about ROOT6 and with what timescale.
- Decide how to proceed with the incorporation of the new look and feel of the website, its implications on the infrastructure as long as the responsabilities involved.
- no consensus; Axel working on a proposal for a different design
- Decide on the organization of the CERN training courses
- Decide if we have enough information, who prepares what slides for the CERN trainings
Result of discussion: * Danilo volunteered to work on the basic course * Lorenzo will work on the analysis one * For the advanced tutorial we have to collect material from all the people
New C++ interfaces
- Decide clear selling arguments for the improved usability and additional functionality.
- Decide the introduction strategy: smooth transition versus the migration all at once, with a tool helping in the migration.
- Decide intermediate deliverables as well as resources and responsibilities.
Result of discussion:
- agreement that ownership and global states are a problem and need to be addressed in future designs.
- Axel proposes to rewrite the histogram subsystem from scratch
- inconclusive where to go; needs dedicated technical discussion
- Decide what is the runtime environment ROOT adopts for multithreaded execution: e.g. TBB, GranDispatch, our thread pool solution or, if we cannot/don't want to decide for some good very concrete technical or strategic reason, an abstract interface to task submission.
- agreement to use TBB for task-based scheduling; configuration at compile-time and run-time whether to enable TBB or not
- no wild card on TBB containers
- We could write down a list of the operations we want to do in parallel or in a thread safe manner, attach a deadline and a person to the delivery of the implied features, always isolating intermediate rough deliverables. For example:
- Transparent interoperability of ROOT and stl threads: TThread-per-thread, TThread::Initialize()
- MultiObject and/or thread local gDirectory
- Decision to rename it into TThreadedObject or MTObject
- Parallel compression with the LZ4 algorighm
- Implicit parallel reading from trees
- Explicit parallel reading from trees, i.e. n user managed threads reading from a tree
- Implicit parallel writing to trees
- Explicit parallel writing from trees, i.e. n user managed threads writing from a tree
- Application of the Process Pool: RooFit, RooStats, TTreeAnalysis
- Agreement on general direction
- Regarding multi-threaded parallelisation, we need to agree on the use case/s to tackle next. In order to make this decision, we should figure out where the big problems are what would be more useful for the users, and give priorities accordingly.
- Decide the evolution of PROOF-lite
- agreement to go for the new implementation.
Input and output (partially treated also in the parallelisation section):
- Decide which technologies to better support in the future, for example SSDs, Kinetic and also how we see the cooperation with DSS and if a potential interplay is possible, e.g. provide native software support for the storage backends which will be provided to users and experiments. As usual, deadlines with intermediate deliverables as well as resources and responsabilities could be discussed.
Decide what is the strategy to support the new readout mode of some experiments, e.g. LHCb in RunIII.
- project for a technical student, e.g. storing TTreeCaches into blobs in Ceph
Interpreter, reflection, core and typesystem:
- Decide a plan for PCMs, realistic intermediate deliverables, realistic coupling with releases (6.6,6.8,6.x?).
- Vassil working on it full time
- no possibility to define a delivery date
- PCMs are not a strictly required/critical feature
- Decide if a type system (plan B) is a component of ROOT 7, by when and the intermediate deliverables.
- Decide if it is realistic to further leverage the llvm technology, for example:
- The CUDA backend
- The Julia frontend
- discussion inconclusive
Graphics, Gui and Visualisation
- Decide about the future of JSROOT and its interplay with the "standard" ROOT graphics. The potential of JSROOT is clear to us (notebooks, cernbox, indico) and is well understood by the users as well. The question is now if we want to keep the current functionality set or if we want it to become the backend of a new interactive visualisation, allowing saving images, resetting to the original plot, change style...
- Bertrand will continue in the direction he is working in now
- If we agree on promoting JSROOT and making it richer in functionalities, what are we going to do about the current divergence between the standard ROOT graphics and JSROOT?
- for now have two parallel lines
- Decide if the graphics UI need some rethinking and/or simplification: leaner interface, helper tools (classes/functions) for plotting, automatic color/line/marker/legend schemes.
- yes, for sure.
- Decide the future of the interface with QT, for example if we want to freeze it or support QT5.
- remove QT subdirectory for ROOT 7
- Provide a specific how to for Qt
- Decide the future of our GUI builder
- frozen for 6; drop for ROOT 7
ROOT as a Service
- Decide what should be core team involvement in the development of the Service
- see last point of "Collaboration and Contributors"
- Decide if and how we commit as a team and as single individuals to ROOT 7
- Decide the strategy to identify the feature set of ROOT 7 as well as the timescales of the project: final delivery, intermediate steps.
- For example: new C++ interfaces, adoption of a task-oriented model, language bindings, I/O formats, Graphics and GUI solutions, Math, ...
- Decide what effort we put in ROOT7 and in ROOT6 and how much we want to evolve ROOT6 and how much we want to dedicate exclusively to ROOT7.
- We could decide how much we see the present work of interface renovation an R&D and to what extent we want to incrementally evolve ROOT 6 according to its findings.
Result of discussion
- timescale for ROOT 7 is end of Run II
- prepare a document with proposed feature-set of ROOT7
- use Wednesday meeting to discuss this with users
Items added during discussion:
- more active follow up on which new platforms to support. Patricia will take care of the technical work of adding machines
- Coding conventions need to be reworked; Philippe will be working on that.