Query & search registries

Find & access data using registries.

Setup

!lamin init --storage ./mydata
Hide code cell output
💡 connected lamindb: testuser1/mydata
import lamindb as ln

ln.settings.verbosity = "info"
💡 connected lamindb: testuser1/mydata

We’ll need some toy data:

ln.Artifact(ln.core.datasets.file_jpg_paradisi05(), description="My image").save()
ln.Artifact.from_df(ln.core.datasets.df_iris(), description="The iris collection").save()
ln.Artifact(ln.core.datasets.file_fastq(), description="My fastq").save()
Hide code cell output
❗ no run & transform get linked, consider calling ln.track()
✅ storing artifact 'DqW6TIJzFUaazoD5Tso6' at '/home/runner/work/lamindb/lamindb/docs/mydata/.lamindb/DqW6TIJzFUaazoD5Tso6.jpg'
❗ no run & transform get linked, consider calling ln.track()
✅ storing artifact 'i02EktimsGzlc3pJRaFx' at '/home/runner/work/lamindb/lamindb/docs/mydata/.lamindb/i02EktimsGzlc3pJRaFx.parquet'
❗ no run & transform get linked, consider calling ln.track()
✅ storing artifact '1kP4KiO3pqgvkkED9sqj' at '/home/runner/work/lamindb/lamindb/docs/mydata/.lamindb/1kP4KiO3pqgvkkED9sqj.fastq.gz'
Artifact(uid='1kP4KiO3pqgvkkED9sqj', description='My fastq', suffix='.fastq.gz', size=20, hash='hi7ZmAzz8sfMd3vIQr-57Q', hash_type='md5', visibility=1, key_is_virtual=True, created_by_id=1, storage_id=1, updated_at='2024-05-25 15:25:45 UTC')

Look up metadata

For entities where we don’t store more than 100k records, a look up object can be a convenient way of selecting a record.

Consider the User registry:

users = ln.User.lookup(field="handle")

With auto-complete, we find a user:

user = users.testuser1
user
User(uid='DzTjkKse', handle='testuser1', name='Test User1', updated_at='2024-05-25 15:25:43 UTC')

Note

You can also auto-complete in a dictionary:

users_dict = ln.User.lookup().dict()

Filter by metadata

Filter for all artifacts created by a user:

ln.Artifact.filter(created_by=user).df()
uid version description key suffix accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
1 DqW6TIJzFUaazoD5Tso6 None My image None .jpg None 29358 r4tnqmKI_SjrkdLzpuWp4g md5 None None 1 True 1 None None 1 2024-05-25 15:25:45.054507+00:00
2 i02EktimsGzlc3pJRaFx None The iris collection None .parquet DataFrame 5629 ah24lV9Ncc8nPL0MumEsdw md5 None None 1 True 1 None None 1 2024-05-25 15:25:45.200058+00:00
3 1kP4KiO3pqgvkkED9sqj None My fastq None .fastq.gz None 20 hi7ZmAzz8sfMd3vIQr-57Q md5 None None 1 True 1 None None 1 2024-05-25 15:25:45.208046+00:00

To access the results encoded in a filter statement, execute its return value with one of:

  • .df(): A pandas DataFrame with each record stored as a row.

  • .all(): An indexable django QuerySet.

  • .list(): A list of records.

  • .one(): Exactly one record. Will raise an error if there is none.

  • .one_or_none(): Either one record or None if there is no query result.

Note

filter() returns a QuerySet.

The ORMs in LaminDB are Django Models and any Django query works. LaminDB extends Django’s API for data scientists.

Under the hood, any .filter() call translates into a SQL select statement.

.one() and .one_or_none() are two parts of LaminDB’s API that are borrowed from SQLAlchemy.

Search for metadata

ln.Artifact.search("iris").df()
uid version description key suffix accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
2 i02EktimsGzlc3pJRaFx None The iris collection None .parquet DataFrame 5629 ah24lV9Ncc8nPL0MumEsdw md5 None None 1 True 1 None None 1 2024-05-25 15:25:45.200058+00:00

Let us create 500 notebook objects with fake titles and save them:

ln.save(
    [
        ln.Transform(name=title, type="notebook")
        for title in ln.core.datasets.fake_bio_notebook_titles(n=500)
    ]
)

We can now search for any combination of terms:

ln.Transform.search("intestine").df().head()
uid version name key description type reference reference_type latest_report_id source_code_id created_by_id updated_at
id
6 FGiqj2FiMc6V None Ascending Colon IgG1 Martinotti cells research... None None notebook None None None None 1 2024-05-25 15:25:49.992348+00:00
13 U0PochPCb41w None Ascending Colon Capillaries Skin rank Cortical... None None notebook None None None None 1 2024-05-25 15:25:49.993465+00:00
14 ARPbiygNLxSW None Red Skeletal Muscle Cell efficiency intestine ... None None notebook None None None None 1 2024-05-25 15:25:49.993623+00:00
17 LwnW4dgzOedg None Crystallin-Containing Lens Fiber Cell IgG Asce... None None notebook None None None None 1 2024-05-25 15:25:49.994100+00:00
23 pdvOUsaZYwvL None Igy IgD IgY intestine. None None notebook None None None None 1 2024-05-25 15:25:49.995063+00:00

Leverage relations

Django has a double-under-score syntax to filter based on related tables.

This syntax enables you to traverse several layers of relations:

ln.Artifact.filter(run__created_by__handle__startswith="testuse").df()
uid version description key suffix accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id

The filter selects all artifacts based on the users who ran the generating notebook.

(Under the hood, in the SQL database, it’s joining the artifact table with the run and the user table.)

Beyond __startswith, Django supports about two dozen field comparators field__comparator=value.

Here are some of them.

and

ln.Artifact.filter(suffix=".jpg", created_by=user).df()
uid version description key suffix accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
1 DqW6TIJzFUaazoD5Tso6 None My image None .jpg None 29358 r4tnqmKI_SjrkdLzpuWp4g md5 None None 1 True 1 None None 1 2024-05-25 15:25:45.054507+00:00

less than/ greater than

Or subset to artifacts greater than 10kB. Here, we can’t use keyword arguments, but need an explicit where statement.

ln.Artifact.filter(created_by=user, size__lt=1e4).df()
uid version description key suffix accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
2 i02EktimsGzlc3pJRaFx None The iris collection None .parquet DataFrame 5629 ah24lV9Ncc8nPL0MumEsdw md5 None None 1 True 1 None None 1 2024-05-25 15:25:45.200058+00:00
3 1kP4KiO3pqgvkkED9sqj None My fastq None .fastq.gz None 20 hi7ZmAzz8sfMd3vIQr-57Q md5 None None 1 True 1 None None 1 2024-05-25 15:25:45.208046+00:00

or

from django.db.models import Q

ln.Artifact.filter().filter(Q(suffix=".jpg") | Q(suffix=".fastq.gz")).df()
uid version description key suffix accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
1 DqW6TIJzFUaazoD5Tso6 None My image None .jpg None 29358 r4tnqmKI_SjrkdLzpuWp4g md5 None None 1 True 1 None None 1 2024-05-25 15:25:45.054507+00:00
3 1kP4KiO3pqgvkkED9sqj None My fastq None .fastq.gz None 20 hi7ZmAzz8sfMd3vIQr-57Q md5 None None 1 True 1 None None 1 2024-05-25 15:25:45.208046+00:00

in

ln.Artifact.filter(suffix__in=[".jpg", ".fastq.gz"]).df()
uid version description key suffix accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
1 DqW6TIJzFUaazoD5Tso6 None My image None .jpg None 29358 r4tnqmKI_SjrkdLzpuWp4g md5 None None 1 True 1 None None 1 2024-05-25 15:25:45.054507+00:00
3 1kP4KiO3pqgvkkED9sqj None My fastq None .fastq.gz None 20 hi7ZmAzz8sfMd3vIQr-57Q md5 None None 1 True 1 None None 1 2024-05-25 15:25:45.208046+00:00

order by

ln.Artifact.filter().order_by("-updated_at").df()
uid version description key suffix accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
3 1kP4KiO3pqgvkkED9sqj None My fastq None .fastq.gz None 20 hi7ZmAzz8sfMd3vIQr-57Q md5 None None 1 True 1 None None 1 2024-05-25 15:25:45.208046+00:00
2 i02EktimsGzlc3pJRaFx None The iris collection None .parquet DataFrame 5629 ah24lV9Ncc8nPL0MumEsdw md5 None None 1 True 1 None None 1 2024-05-25 15:25:45.200058+00:00
1 DqW6TIJzFUaazoD5Tso6 None My image None .jpg None 29358 r4tnqmKI_SjrkdLzpuWp4g md5 None None 1 True 1 None None 1 2024-05-25 15:25:45.054507+00:00

contains

ln.Transform.filter(name__contains="search").df().head(10)
uid version name key description type reference reference_type latest_report_id source_code_id created_by_id updated_at
id
4 9f62oh0gWlAG None Research IgG1 IgD IgG1 IgG3 IgM IgG1 IgY. None None notebook None None None None 1 2024-05-25 15:25:49.992005+00:00
6 FGiqj2FiMc6V None Ascending Colon IgG1 Martinotti cells research... None None notebook None None None None 1 2024-05-25 15:25:49.992348+00:00
12 gQ8Yd8sInbRW None Igd research IgE IgY Martinotti cells classify... None None notebook None None None None 1 2024-05-25 15:25:49.993306+00:00
16 ZCMK5FlIpmjb None Research Skin IgY IgG1 result Chandelier cells. None None notebook None None None None 1 2024-05-25 15:25:49.993940+00:00
19 TQ3Hr5adK5cF None Igy IgE IgE Cuticular Martinotti cells Cortica... None None notebook None None None None 1 2024-05-25 15:25:49.994420+00:00
35 lqRJT6vU5IUW None Study research intestinal. None None notebook None None None None 1 2024-05-25 15:25:49.997102+00:00
48 2qM7jjtfOWpi None Igd research Red skeletal muscle cell IgG IgG3... None None notebook None None None None 1 2024-05-25 15:25:49.999176+00:00
67 VJTOSvYLGsHa None Research Martinotti cells result cluster IgG I... None None notebook None None None None 1 2024-05-25 15:25:50.002225+00:00
69 kOFzxfz8vvg9 None Igd IgG IgA research cluster Ascending colon I... None None notebook None None None None 1 2024-05-25 15:25:50.002546+00:00
93 BV4Zr9eboy3W None Research neurotensin IgD. None None notebook None None None None 1 2024-05-25 15:25:50.009290+00:00

And case-insensitive:

ln.Transform.filter(name__icontains="Search").df().head(10)
uid version name key description type reference reference_type latest_report_id source_code_id created_by_id updated_at
id
4 9f62oh0gWlAG None Research IgG1 IgD IgG1 IgG3 IgM IgG1 IgY. None None notebook None None None None 1 2024-05-25 15:25:49.992005+00:00
6 FGiqj2FiMc6V None Ascending Colon IgG1 Martinotti cells research... None None notebook None None None None 1 2024-05-25 15:25:49.992348+00:00
12 gQ8Yd8sInbRW None Igd research IgE IgY Martinotti cells classify... None None notebook None None None None 1 2024-05-25 15:25:49.993306+00:00
16 ZCMK5FlIpmjb None Research Skin IgY IgG1 result Chandelier cells. None None notebook None None None None 1 2024-05-25 15:25:49.993940+00:00
19 TQ3Hr5adK5cF None Igy IgE IgE Cuticular Martinotti cells Cortica... None None notebook None None None None 1 2024-05-25 15:25:49.994420+00:00
35 lqRJT6vU5IUW None Study research intestinal. None None notebook None None None None 1 2024-05-25 15:25:49.997102+00:00
48 2qM7jjtfOWpi None Igd research Red skeletal muscle cell IgG IgG3... None None notebook None None None None 1 2024-05-25 15:25:49.999176+00:00
67 VJTOSvYLGsHa None Research Martinotti cells result cluster IgG I... None None notebook None None None None 1 2024-05-25 15:25:50.002225+00:00
69 kOFzxfz8vvg9 None Igd IgG IgA research cluster Ascending colon I... None None notebook None None None None 1 2024-05-25 15:25:50.002546+00:00
93 BV4Zr9eboy3W None Research neurotensin IgD. None None notebook None None None None 1 2024-05-25 15:25:50.009290+00:00

startswith

ln.Transform.filter(name__startswith="Research").df()
uid version name key description type reference reference_type latest_report_id source_code_id created_by_id updated_at
id
4 9f62oh0gWlAG None Research IgG1 IgD IgG1 IgG3 IgM IgG1 IgY. None None notebook None None None None 1 2024-05-25 15:25:49.992005+00:00
16 ZCMK5FlIpmjb None Research Skin IgY IgG1 result Chandelier cells. None None notebook None None None None 1 2024-05-25 15:25:49.993940+00:00
67 VJTOSvYLGsHa None Research Martinotti cells result cluster IgG I... None None notebook None None None None 1 2024-05-25 15:25:50.002225+00:00
93 BV4Zr9eboy3W None Research neurotensin IgD. None None notebook None None None None 1 2024-05-25 15:25:50.009290+00:00
107 SEjT5K0eLr0N None Research IgG1 rank IgY intestinal cluster clas... None None notebook None None None None 1 2024-05-25 15:25:50.011436+00:00
189 nZF4skfiEGkA None Research Smooth muscle cell IgG IgE. None None notebook None None None None 1 2024-05-25 15:25:50.026554+00:00
297 wJKDKLAeQkLp None Research candidate Cortical study. None None notebook None None None None 1 2024-05-25 15:25:50.045752+00:00
349 QBdsx4ytUfrx None Research rank neurotensin IgG1. None None notebook None None None None 1 2024-05-25 15:25:50.056321+00:00
353 QRK9JpEGa6XA None Research Smooth muscle cell IgG3 IgG4 IgM IgG ... None None notebook None None None None 1 2024-05-25 15:25:50.056940+00:00
Hide code cell content
# clean up test instance
!lamin delete --force mydata
!rm -r mydata
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.9/x64/bin/lamin", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/rich_click/rich_command.py", line 367, in __call__
    return super().__call__(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/rich_click/rich_command.py", line 152, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/lamin_cli/__main__.py", line 103, in delete
    return delete(instance, force=force)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/lamindb_setup/_delete.py", line 98, in delete
    n_objects = check_storage_is_empty(
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/lamindb_setup/core/upath.py", line 798, in check_storage_is_empty
    raise InstanceNotEmpty(message)
lamindb_setup.core.upath.InstanceNotEmpty: Storage /home/runner/work/lamindb/lamindb/docs/mydata/.lamindb contains 3 objects ('_is_initialized' ignored) - delete them prior to deleting the instance
['/home/runner/work/lamindb/lamindb/docs/mydata/.lamindb/1kP4KiO3pqgvkkED9sqj.fastq.gz', '/home/runner/work/lamindb/lamindb/docs/mydata/.lamindb/DqW6TIJzFUaazoD5Tso6.jpg', '/home/runner/work/lamindb/lamindb/docs/mydata/.lamindb/_is_initialized', '/home/runner/work/lamindb/lamindb/docs/mydata/.lamindb/i02EktimsGzlc3pJRaFx.parquet']