A join on namesake secondary attributes raises the stale-~lineage error on a freshly declared schema #1640

Summary

On a schema declared entirely by the current release, a join on two tables that each declare a secondary attribute with the same name raises the stale-lineage error:

Cannot join on attribute `input_hash`: lineage missing on one side (None vs None). This usually indicates a stale `~lineage` entry from an older DataJoint version or an incomplete declare. Run `schema.rebuild_lineage()` ...

Nothing is stale. ~lineage holds a row for every primary-key and foreign-key attribute. A native secondary attribute has no lineage by design, so its lineage is None on both sides. schema.rebuild_lineage() writes the same rows, and the error persists. Computed attributes of two projections raise the same error. The message tells users to repair a healthy schema, with a repair that cannot change the outcome.

The None branch in assert_join_compatibility (condition.py:271-284, added in #1467 for #1454) treats any None lineage as a missing ~lineage row. But None is also the lineage of every native secondary attribute and every computed attribute: Table._populate_lineage (table.py:247-276) and rebuild_schema_lineage (lineage.py:366-367) write rows only for primary-key and foreign-key attributes. The check at decoration (schemas.py:321) already makes this distinction, since it treats None as a missing row only for primary-key attributes.

Reproduction

DataJoint 2.3.5.dev8 (master, a00b89f), MySQL 8.0.46, Python 3.11, on a new schema:

import datajoint as dj

schema = dj.Schema("scratch_lineage_secondary")


@schema
class Image(dj.Manual):
    definition = """
    image_id : int32
    """


@schema
class Segmentation(dj.Manual):
    definition = """
    -> Image
    ---
    input_hash : varchar(64)
    cell_count : int32
    """


@schema
class Profile(dj.Manual):
    definition = """
    -> Image
    ---
    input_hash : varchar(64)
    mean_area : float32
    """


print("~lineage:", schema.connection.query("SELECT * FROM `scratch_lineage_secondary`.`~lineage` ORDER BY 1").fetchall())
print("input_hash lineage:", Segmentation.heading["input_hash"].lineage, Profile.heading["input_hash"].lineage)

for attempt in ("fresh schema", "after rebuild_lineage()"):
    try:
        Segmentation * Profile
    except dj.DataJointError as e:
        print(f"{attempt}: {e}")
    schema.rebuild_lineage()

try:
    Segmentation.proj(n="cell_count + 1") * Profile.proj(n="mean_area * 2")
except dj.DataJointError as e:
    print(f"computed attributes: {e}")

Output:

~lineage: (('image', 'image_id', 'scratch_lineage_secondary.image.image_id'), ('profile', 'image_id', 'scratch_lineage_secondary.image.image_id'), ('segmentation', 'image_id', 'scratch_lineage_secondary.image.image_id'))
input_hash lineage: None None
fresh schema: Cannot join on attribute `input_hash`: lineage missing on one side (None vs None). This usually indicates a stale `~lineage` entry from an older DataJoint version or an incomplete declare. Run `schema.rebuild_lineage()` to recompute lineage from current FK definitions. If the lineages are genuinely different, use `.proj()` to rename one of the attributes.
after rebuild_lineage(): Cannot join on attribute `input_hash`: lineage missing on one side (None vs None). This usually indicates a stale `~lineage` entry from an older DataJoint version or an incomplete declare. Run `schema.rebuild_lineage()` to recompute lineage from current FK definitions. If the lineages are genuinely different, use `.proj()` to rename one of the attributes.
computed attributes: Cannot join on attribute `n`: lineage missing on one side (None vs None). This usually indicates a stale `~lineage` entry from an older DataJoint version or an incomplete declare. Run `schema.rebuild_lineage()` to recompute lineage from current FK definitions. If the lineages are genuinely different, use `.proj()` to rename one of the attributes.

Dropping and re-declaring tables, deleting and recomputing rows, and alter() on attribute types all leave ~lineage intact; none of them is needed to reach this error.

Expected behavior

• On a schema declared by this version, a join never raises the stale-lineage error and never suggests rebuild_lineage().
• A join on namesakes that are not homologous, such as native secondary attributes of two tables or computed attributes of two projections, is refused with an error that names the attribute, says it is not inherited through a foreign key, and suggests .proj() to rename or exclude it on one side.

Either of two changes would give this:

1. Record lineage for native secondary attributes at declaration and in rebuild_schema_lineage, as schema.table.attribute, as for native primary-key attributes. A table attribute's lineage is then None only when its row is missing, the stale-lineage message is accurate whenever it appears, and this case gets the existing "different lineages" error.
2. Keep native secondary attributes without lineage, and raise the stale-lineage message only for attributes that must have a row (a table's primary-key and foreign-key attributes). Every other None gets the non-homologous error.