Skip to content

Problems running the tutorial on glados #1

Open

Description

@JinZhou5042

I went through Modules 1-9 as a participant would, on glados with conda, for both coffea and RDataFrame.

Tested with Python 3.14, awkward 2.14.0, ROOT 6.40, vine-reduce 2026.10.3, ndcctools 7.17.5, workers from the glados condor pool.

1. glados PYTHONPATH breaks the conda env (Module 1)

glados sets PYTHONPATH to the system Python 3.9 packages in every login shell (/etc/profile.d/settings.sh). Because of that, conda env create fails in the pip step with permission denied under /usr/lib64/python3.9. If an env does get built, it picks up the system numpy and Module 2 fails, even though the Module 1 import check prints ok.

2. awkward pin conflicts with coffea (Module 1)

awkward=2.8.7 in environment.yml can't be solved together with coffea>=2025.10.2. Also, vine-reduce>=2026.9.8 allows older versions whose Module 9 output differs from the docs.

3. pixi isn't available on glados (Module 1)

Option B runs pixi install, but pixi isn't installed on glados (pixi: command not found), and Module 1 doesn't say how to install it.

4. Generated files aren't git-ignored (Module 1)

Module 1 says they're ignored, but factory.json, plots/, vine_reduce-envs/ and with_oasis_certs show up as untracked.

5. Timings are optimistic on glados (Modules 1, 3)

Env creation took about 60 minutes (docs say up to 15), and Module 3 took several minutes instead of the ~1.5 / 4 minutes in the docs. Most of it is importing from NFS.

6. Wrong paths in datasets_vast.json (Modules 2, 3, 6, 9)

The paths use /project01/ndcms/btovar/store/user/btovar/tutorial/..., which doesn't exist. The files are under /project01/ndcms/btovar/tutorial/tutorial/..., as the Appendix says.

7. resource_monitor crashes (Modules 4, 5, 7, 8)

With ndcctools 7.17.5 the monitor sometimes dies with the itable iteration has not been reset since last modification, and the task comes back as OUTPUT_MISSING. Retries hide most of these, but in Module 8 one chunk failed all its attempts and the whole run aborted.

In resource_monitor.c, cleanup_zombies() removes entries from the processes table while iterating over it, and the SIGCHLD handler iterates over the same table while the main loop is iterating.

8. Workers are slow to connect, or never connect (Modules 4, 5, 7, 8)

Condor starts the workers right away, but they can take several minutes to connect to the manager, and some give up. The manager advertises all of glados's addresses to the catalog, including a link-local one (169.254.1.2) that the pool can't reach. If a worker tries that address first, it waits there until it times out. Stale catalog entries from earlier runs under the same name make it worse.

9. Wrong cortado reference in Module 5

The docs point to examples/scale-cortado/vr_cortado.py and describe memory settings it doesn't have. The file is examples/cortado/vr_cortado.py, and it only sets cores.

10. Module 6 checkpoint shows up later than the docs say

The first checkpoint appeared after ~40 s, not ~20 / 30 s. The final result is correct.

11. Users can't write the skim directory (Module 7)

The skim scripts write to /project01/ndcms/users/{user}/trijet-skims, but /project01/ndcms/users isn't group-writable, so users get PermissionError.

12 Broken links in Module 10

https://cclnd.cse.nd.edu doesn't load, and the three GitHub links start with https://git@github/ instead of https://github.com/.

14. Users need to be in osguser (Appendix)

/project01/ndcms is only readable by the osguser group, but the Appendix says the data is world-readable. Users outside the group can't read the data or write skims.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions