Quantum.jl Specifics: Structure of Base Repo

[jrevels]

Edit: See below discussion; this issue is now about the specifics of structuring a base quantum repo separate from my own (now called QuDirac.jl).

Hi all,

I just finished the documentation for Quantum.jl, which provides descriptions and examples of the package's current functionality. These docs can be found here.

I am working separately on an API that will provide more specific information on all the functions implemented by Quantum.jl. While the API isn't yet finished, I feel the current documentation linked above is comprehensive enough to get a feel for the package's goals and behavior.

I'm hoping that Quantum.jl will provide a lot of the foundational functionality necessary to start doing quantum mechanics in Julia, as well as a solid framework for the development of type representations of quantum objects. I'm starting this issue so that everybody can go and read the docs I've written up, and see whether you all are in favor of Quantum.jl becoming the foundational package for JuliaQ (i.e. other quantum libraries will likely rely on it as a dependency).

Any questions or concerns about the package can be presented in this issue, but if you have more specific ideas or detailed concerns, I suggest opening up an issue in Quantum.jl's issue tracker instead of JuliaQ's. That way, we can keep development of Quantum.jl all in one place. I'm not the most amazing Julia programmer in the world, so I hope that others will contribute bug fixes, optimization knowledge, and new features to make the package safer, faster and more useful to the community.

If all goes well, I would like to officially announce Quantum.jl's release on the julia-dev/julia-users google groups so that we can let the community know that it exists and get more people interested in the project.

---

Here's a FAQ for what I anticipate to be common questions for this issue:

Wow, that code base sure is messy...
The code base is fairly disorganized at the moment. I've been scrambling the past couple days to get everything in place, so there's still a bit of cleaning up to do. I plan on doing a good bit of reorganization in addition to working on the API this week.

What about states/operators in continuous bases?
Right now, all the functionality accompanying states and operators assumes that one is working with discrete bases. This is simply because of the work that I was doing that prompted me to build Quantum.jl in the first place. Now, this isn't actually as big of a problem as it might sound - the type representations I've implemented could certainly be extended to function efficiently on continuous bases without adversely affecting or interfering with the existing discrete functionality in any way.

Could Quantum.jl's DiracMatrix and DiracVector types work with other, existing algorithms/libraries?
YES. Algorithms or code that works with Julia's SparseMatrixCSC can be extended to DiracMatrix or DiracVector objects, since coefficients arrays are stored as their own field. In fact, if we decided to change the array type to something other than a SparseMatrixCSC (perhaps to make use of Julia's matrix-specfic types, like Hermitian), the code base could be refactored to support that new implementation without a huge amount of effort. This also means that Quantum.jl could take advantage of interfacing with more mature libraries like QuTiP.

Who are you?
I'm a recent graduate (B.Sc.) who double majored in Computer Science and Physics at American University in Washington, DC. I did undergraduate research in representation theory as it applies to modal entanglement generation, and used Julia to do most of the computational work that I ran into. A lot of Quantum.jl was based on my desire to have a system for Julia that would allow the calculations I was doing to be more idiomatically transferred from the blackboard to the computer.

---

P.S. I figured this issue should go here instead of the Roadmap repo, but let me know if I should move it and I will.

[i2000s]

I was encouraging Jarrett to present his work to everyone here, because I think, among other works I can find, his project can become a base for JuliaQ to plug in more advanced functions. To my understanding, our quantum projects is better to have some basic types and methods that other more advanced functions can build on. Quantum.jl has already built in some basic types like quantum states and quantum operators, as well as some tensor operations with some optimized algorithms (sparseMatrix method and so on). I know a few other people have their codes working as well, we can compare if you would like to present along this line. Jarrett has provided a detail document for everyone to check the functionality and go over the source code. I think discussions regarding how can we make a good starting point for JuliaQ project should go here. We can discuss more detailed technical details later--when our JuliaQ project starts after some discussions and agreements.

To people who are not familiar with Julia organizations, the purpose of building up this JuliaQ umbrella organization (like many others) is to bring together people from different yet related areas to carefully think about building a few fundamental quantum libraries in Julia that can span into helpful tools for people in this broad field to use. In my biased opinion, it would be helpful to start with basic definitions and some typical solvers for quantum dynamics, and then more specific branches of quantum theory can be easily built upon or in separated project with different conventions. I have discussed the long term roadmap and near future directions with a few people in the community, and have come out the biased roadmap on the roadmap issue page. I am still encouraging everyone to bring about their own ideas and existing projects here. A few of my colleagues in Quantum Information community will join us soon as well. I think we will be able to figure out a way to integrate different thoughts and codes as a JuliaQ project. Quantum.jl is just an example to present here for us to judge if it is a good base to work on as a JuliaQ project. I am not saying Quantum.jl is a perfect work, but it can lead us to start discussing what fundamental types of object do we need, what is a good foundation for quantum libraries to have dynamics algorithms, etc. We may need to have superoperaters, graph/cluster states and so on. If you have already did similar fundamental things, or have some better ideas, we should compare and discuss.

My hope is that eventually, we can find out some starting point to build our consistent quantum libraries in Julia, which everyone with different specialties of quantum theory can contribute and benefit from. I have proposed a process of organizing JuliaQ projects (see the related issue, and it is still open for discussion). If people support to have Jarrett's Quantum.jl project or any other project as a base to start JuliaQ projects, and Jarrett or any other author agrees to move his/her personal project to becoming a JuliaQ community project, we shall be able to implement or integrate more features in different branches or repos of the fundamental JuliaQ project. If one just wants to develop his/her own project for self-satisfaction, surely he/she has the freedom, and shall not bring the project to the Julia umbrella organization. If no one wants to transfer their own project as a JuliaQ organization project, we will still be able to start our community project from the scratch through discussions and plugging in open-source contributions among the community. I appreciate your contribution for the good hope. Thanks.

PS: I will be away to prepare for a conference for a couple of weeks, and discuss more with people in the Julia community. I will be back to make contributions to the code repo soon (if no one do the Quantum Monte Carlo part, I will do it). I believe the organization has already had enough member to run smoothly by self-organization.

Cheers,
Qi

[jrevels]

So, it doesn't seem like people have voiced any concerns with the package yet. If everyone really is fine with it, I'm going to move it into the JuliaQ repo and announce it myself in the julia-dev and julia-users next Monday, basically with the same FAQ I've posted here (without the "messy codebase" part, since I hope to have everything cleaned up by then). If anybody is opposed to this, feel free to comment so we can discuss it. Otherwise, I'm going to go through with releasing it since I wish to keep up the pace of development for the project.

[i2000s]

Feel free to move it to JuliaQuantum repo at any time. Maybe wait for another week for release? New members are joining us...

[acroy]

@jrevels: I am very interested in your package and will try to have a thorough look into it over next couple of days, but currently I am too tied up with other things. This shouldn't prevent you to announce and release your package if you want to get a wider audience. If you tag it as 0.0.1 or such it is clear that it is WIP, so you don't have to worry if much if something is not perfect yet. In contrast, you might get valuable feedback.

Regarding moving it to JuliaQuantum I am not really sure yet what the best strategy is. I have the impression that developing a package in a private repo (in contrast to an organization repo) has the advantage that one is more flexible and it might be easier to try new things in the beginning, also simply because there is only one who decides. Later, when the package is more mature it makes perfect sense to maintain it by an organization. I think the latter is most useful for general discussions, defining common interfaces and such, but as the discussion on JuliaGraphics shows there are different approaches and opinions about this.

[Jutho]

@jrevels : I too will try to look at package before the end of the weekend and think about how this would fit in or combine with the kind of code I have been working in. I will also try to share some info about what I am working on and what I hope I will be able to contribute (although it won't be for the first few weeks).

[jrevels]

If people are still checking it out, I'll refrain from an official announcement for a couple days. I definitely would like to release it by the end of this upcoming week, as I agree with @acroy that a W.I.P. initial release could lead to useful feedback early on. If folks decide they don't want to use this package for foundational work, I'll change it's name to something else so that the name "Quantum.jl" can be used elsewhere.

Insofar as moving the repo goes, I'm fine with whatever is decided by the community. I do see the advantages of letting it sit where it is until it has reached a certain level of maturity, but I don't really have enough experience yet with this kind of thing to make a decision myself. Either way, once people are comfortable with the package, I'd like to add one or two volunteers on as admins so that my availability won't be the bottleneck for further development. Otherwise, if nobody steps up, I'll continue to maintain the repo myself, which will at least ensure that everything remains consistent as development moves forward.

[jrevels]

So, I came up with a basic idea for how we might structure the main repo for more
modularity and wider scope than just the representation-centric work I've done.

It'd look something like this:

Quantum.jl/
    doc/
    src/
        representations/
            (module for representation work)
        dynamics/
            (module for dynamics work)
        visualization/
            (module for visualization work)
        .
        .
        .
        Quantum.jl
    test/
    REQUIRE
    LICENSE.md
    README.md

What we actually include in src/ might differ, of course (see last paragraph), but the code I've written would go under something like representations/ in the above heirarchy. Code on other topics would be filed under separate folders and in separate modules, and then the Quantum.jl file will act as a wrapper that incorporates all the other modules.

This way, I'd feel a little better about releasing the whole thing earlier as my code would very obviously only consitute a single module of a larger project. Also, this new structure is more contributor-friendly, so it would hopefully encourage more people not already in the JuliaQuantum group to collaborate.

As for what topics Quantum.jl will cover, we should probably draw the line between "foundational" and "specialized" sooner rather than later. This way we can begin to divy up functionality between Quantum.jl and auxiliary packages. #1 was opened up for this purpose, so feel free to discuss this there.

[acroy]

I don't think we should merge everything into a super-repo. At least I don't see any necessity to do that. Let's rather keep different functionality in separate repos (ideally under JuliaQuantum coordination). Having an organization will (hopefully) make sure that all the package work nicely together and have a certain standard.
Since we can specify dependencies for each package (via REQUIRE), we can effectively realize the hierarchy @jrevels proposes on a package level. We can also have meta-packages, which pull in all or a subset of packages in JuliaQuantum. Quantum.jl might become one of the base packages on which (most) other packages will depend on. But I guess we should wait a bit to see which packages exist and how they fit into a bigger scheme.

@jrevels : You shouldn't feel bad about releasing your package now or at any time that suits you. Others might have thought about/implemented a similar functionality but in a different context. Getting their input would be really valuable (you don't necessarily have to register the package now, you can also just announce it on julia-users since Pkg.clone makes it easy to try out).

[i2000s]

@acroy: Regarding the merge of packages, I think it all depends on what
scheme can we implement in the near future. For example, advanced topics
like quantum chemistry and quantum information will require both quantumRep
and qauntumDynamics (Master equation, Stochastic equation solver and so on)
at the same time. If we can just merge the representation and some commonly
used dynamics solver in one package, it will be easier for those
developments along advanced topics. People with different strong suits on
specific coding aspects will be able to monitor the change of the
algorithms and dependency easier in a combined repo, and maintain
consistency and performance in a better way. We shall see if we need this
kind of combination as more utilities plugged in, and would be easy to
merge at an earlier stage if we can predict this need. In any case, if you
decide to include the quantum.jl functionality into your ME solver, or just
want to use a part of them yet to define your own types, you can do it, and
I think it is already there. The thing is that this will reduce the
dependency and consistency checking work, which is a necessary part to
guarantee quality; it may also increase the work for testing and
communication if anything is changed in an isolated repo. If you decide to
completely use the quantum.jl as the basis for the dynamic solvers, why not
just combine these two into one repo rather than calling another package
every time you run the dynamics solver. That will help the more advanced
developments. Any way, we can decide this later as things become more
clear.

Surely, we'd better not to make a super big repo which includes everything
of quantum. That will be frustrating for too many details of conflicts.

On Tue, Jul 22, 2014 at 2:27 PM, acroy notifications@github.com wrote:

I don't think we should merge everything into a super-repo. At least I
don't see any necessity to do that. Let's rather keep different
functionality in separate repos (ideally under JuliaQuantum coordination).
Having an organization will (hopefully) make sure that all the package work
nicely together and have a certain standard.
Since we can specify dependencies for each package (via REQUIRE), we can
effectively realize the hierarchy @jrevels https://github.com/jrevels
proposes on a package level. We can also have meta-packages, which pull in
all or a subset of packages in JuliaQuantum. Quantum.jl might become one
of the base packages on which (most) other packages will depend on. But I
guess we should wait a bit to see which packages exist and how they fit
into a bigger scheme.

@jrevels https://github.com/jrevels : You shouldn't feel bad about
releasing your package now or at any time that suits you. Others might have
thought about/implemented a similar functionality but in a different
context. Getting their input would be really valuable (you don't
necessarily have to register the package now, you can also just announce it
on julia-users since Pkg.clone makes it easy to try out).

—
Reply to this email directly or view it on GitHub
#4 (comment)
.

[jrevels]

At this point I'm inclined to agree with @acroy. I think the push towards a broad-ranging repo was an idea carried over from the development of python packages like QuTiP, but other Julia organizations like JuliaOpt and JuliaStats have already established the precedent of a more separated structure, and have met with a high degree of success. Consistency/cross-compatibility could be managed in the overall organization issue tracker, and individual packages could manage themselves for more rapid development. It also allows others to only list the functionality they need as dependencies.

In regards to the work I've done myself, I'm actually starting to think it would be more beneficial to go ahead and release it as "QuReps.jl" or "Dirac.jl". My work stores and manipulates abstract states overtop of representations, which allows a lot of cool things to be done, but the complexity of this kind of framework could be unnecessarily high for people who don't need these features. In particular, often times only manipulation of a representation is needed, and the basis is easily assumed. In that case, unless one wanted to perform certain kinds of analysis, my package wouldn't really be relevant to them. That's why I was hesitant about pushing it as "QuantumBase.jl", and made my comment about modularity above - I just wasn't sure whether the features I was providing constituted a ubiquitously usable toolset.

However, I do think my work is far enough along that I have no problems registering it as a package.

@acroy I really would like to feature QuDOS.jl as a dependency in a REQUIRE file, because I see a few places where my code could be improved by calling functions from QuDOS.jl. For this reason, I would appreciate if you could register QuDOS.jl with the package manager when you get the chance. If you feel it's not ready, however, I understand and will just instruct users of QuReps.jl to manually install it using Pkg.clone.

I also vote that "Quantum.jl" be a wrapper package for several other useful packages, like mine and @acroy 's. That way, as he said, we can still have the same hierarchy I talked about but with more modularity.

[i2000s]

If you still think keeping things separated is a success of Julia, you may
want to check this issue.
On that issue, my understanding is that there is a change in the definition
of equal and = in the Julia base. No one realized the impact of the change
at the beginning, later on people identified this issue and manually filed
100+ issues on isolated repos depending on this equal op. From May to
today, there are still many people taking charge of separated packages not
responding the filed issue, which to some extent hindered the release of
the Julia V0.3. This has been a important lesson back in my mind when I
thought of starting this organization. To avoid this kind of issues due to
over separated management, I think we should combine highly dependent
functions in one place. If anything is changed, one can simply go over the
test script in the repo to see if there is any conflict or error for all
dependent functions.

For our case, I guess every package for quantum will need a representation
basis. Any advanced topics will likely to need a quantum dynamics solver. A
base with only representation will be hardly useful for specific
applications. A dynamic solver without using representation is impossible.
These two aspects of quantum will form a good foundation for all other high
level developments. I think we can combine some of the commonly used rep
and dynamics into one to reduce the overlap works and to maintain a high
quality of code library. People from different areas can focus on optimize
this basis to the best, and monitor the consistency and completeness in a
deep way. For example, Jarrett has been thinking of representations of
quantum in Julia for a long time, the integration of some common functions
into the original quantum representation base will be benefited from the
optimized code of the quantum representation base that Jarrett has devised
first. This will booster the efficiency of other people's work in a long
run. I have also considered the friend repo mechanism which allows people
don't want to be coordinated under the quality control standard of the
JuliaQuantum community to be forked here and get benefits from the
community. I think we can discuss this in more details as more concerns
come later in a separated issue.

@jrevels: Feel free to release your package in any form at any time in any
name. What I am saying about the release cycle is for the JuliaQuantum
charged projects for the sake of quality and reputation. Now, we haven't
decided to make quantum.jl as the base for the community to work together.
I think you did a good job on it.

On Tue, Jul 22, 2014 at 3:35 PM, Jarrett Revels notifications@github.com
wrote:

At this point I'm inclined to agree with @acroy https://github.com/acroy.
I think the push towards a broad-ranging repo was an idea carried over from
the development of python packages like QuTiP, but other Julia
organizations like JuliaOpt https://github.com/JuliaOpt and JuliaStats
https://github.com/JuliaStats have already established the precedent of
a more separated structure, and have met with a high degree of success.
Consistency/cross-compatibility could be managed in the overall
organization issue tracker, and individual packages could manage themselves
for more rapid development. It also allows people to only depend on the
functionality they need.

In regards to the work I've done myself, I'm actually starting to think it
would be more beneficial to go ahead and release it as "QuReps.jl". My work
stores and manipulates representations explicitly, which allows a lot of
cool things to be done, but the complexity of this kind of framework could
be unnecessarily high for people who don't need these features. That's why
I was hesitant about pushing it as "QuantumBase.jl", and made my comment
about modularity above - I just wasn't sure whether the features I was
providing constituted a ubiquitously usable toolset.

However, I think do think my work is far enough along that I have no
problems registering it as a package.

@acroy https://github.com/acroy I really would like to feature QuDOS.jl
as a dependency in a REQUIRE file, because I see a few places where my code
could be improved by calling functions from QuDOS.jl. For this reason, I
would appreciate if you could register QuDOS.jl with the package manager
when you get the chance. If you feel it's not ready, however, I understand
and will just instruct users of QuReps.jl to manually install it using
Pkg.clone.

—
Reply to this email directly or view it on GitHub
#4 (comment)
.

[jrevels]

You make a good point, but I'm not convinced that adhering to a "separation of concerns" philosophy inherently translates to a sacrifice in quality assurance. In fact, I would be disposed to argue the opposite; modularity makes it easier to isolate, delegate, and in general, respond to issues quickly.

As for testing: If we choose to go the route of making Quantum.jl a wrapper repo, we can still write a centralized test suite that checks cross-package functionality, while allowing for package-specific source code and module-specific tests to be written in individual packages.

The == vs. isequals problem isn't a problem about how developers have chosen to organize packages, but rather an issue in Julia's base source. I honestly don't think the response would've been that different if people had used more centralized repos instead of the structure that currently exists. If anything, I imagine that it would've been even more difficult to respond to that particular problem if packages had been more centralized, since the problem probably manifested itself differently for each package. Every package needed to address the issue of how it overloads equality operators in a manner specific to the functionality that package is trying to provide. This would be a more difficult task if developers had not logically separated out packages by topic.

For our case, I guess every package for quantum will need a representation basis.

But not necessarily every package will need to store basis states explicitly, as my work does. As an example, I'm pretty sure nothing in QuTiP explicitly stores basis states - it assumes/implies them, as one normally does when doing general vector calculations. I'm not trying to say my package won't be useful for foundational work or as a dependency - I really, really would like it to be - I'm just saying that some people probably really do only need a certain dynamical solver, or a visualization tool, etc. If that's the case, they could just use what they need. If instead they wanted a whole suite to use, they could just use our wrapper repo, Quantum.jl, which would make sure all relevant packages are installed and functioning together properly.

The structure of the "ecosystem" of packages that has arisen since Julia started is a direct result of how Julia's package management system was designed. I think trying to build a "super-repo", as @acroy put it, would be working against that design to our detriment.

I'm willing to hear out arguments to the contrary, and hope I'm not coming across as combative or anything. I just feel like this is a discussion worth having, and that it's the appropriate time to have it.

[acroy]

I fully agree with @jrevels, but let me add one or two comments on this. 1) We should be aware that Julia and all packages are still moving targets (pre 1.0), so issues like == vs isequal might happen (although I think this issues also resulted partly from confusion of the two equalities in the first place). Since our packages will be hosted by JuliaQuantum, there will be several people who can make sure that issues are addressed timely.
2) Julia has a well designed package system and now we also have tools like PkgEvaluator.jl to monitor the quality of packages. We should take advantage of that.

In addition to that I think it would simply be impractical to merge packages with different functionality. For example, a QuantumViz package (for visualization) would need QuantumStates but wouldn't care about any dynamics (and vice versa). Also quantum chemistry or electronic structure packages would care little about Lindblad QMEs. IMHO using packages and dependencies is a viable way to make sure everything works together while allowing to focus on the respective functionality. That doesn't mean that there shouldn't be a transfer of functionality between packages. For instance, purity, fidelity, negativity and such could probably be in some other package. But this kind of transfer can be easily coordinated across the organization.

[i2000s]

I think everyone is on a right track. As long as we can easily resolve these organizational problems, I am perfectly fine with any form of organizing projects. I think for this stage, we need to figure out what we can do to work out some JuliaQuantum project, what's the hierarchy structure of our projects... Talk to you guys later.

[acroy]

@jrevels : I see your point about storing the basis. Most numerical codes will only care about the coefficients (like in my case) and implicitly assume a basis. The question is how those "coefficient vectors" fit into a more general type hierarchy like yours (or @Jutho's)? For now we can just convert between the types, but eventually we probably want to have some common type(s), which can be used by different packages.

[jrevels]

This is an old issue - all discussion potentially going here should be filed as an issue on the QuBase.jl project page.